I would like to find out what is the e2e automation test coverage to be able to identify the gaps in our test suite and have a quick insight into the amount of coverage. What is the best way or tool to use?
What type of coverage are you asking for? Code coverage? Feature coverage? How important is the depth?
One way would be (which I do frequently as I develop the automation):
- Learning the code and creating a mapping to the product what is covered.
Code coverage can be done by some tools (I haven’t done this).
Cypress has some of the best docs I have seen on E2E UI code coverage - Code Coverage | Cypress Documentation. Most of the other tools will follow a similar approach.
This isn’t a silver bullet though as just because you hit a statement/branch it doesn’t mean you did anything useful. It doesn’t check that you are asserting the correct things.
It can help identify any gaps when your UI behaves differently based on properties, backed responses, error scenarios etc.
I would need to identify the gaps in the e2e automation suite in terms of feature coverage. What tool do you use to do the mapping?
I might also look into code coverage and the depth in the near future.
I don’t have any specific tools aside something where I write tables or lists.
Make a list of feature + depth of the product you care for and then look into the automation code and note what you finde.
Your main tools is your brain. Your skill.
Hello, and welcome!
You want to know stuff about your product, probably so you can feel better that you’re releasing it without something going wrong. You want to do that so that you don’t annoy customers, so you can continue enjoying their money.
To do that you are using a tool that programmatically checks some facts, and tells you if the result differs from one that you told it about.
Your goal, as I see it, is to take the set of facts you want checked, and compare it to the facts you actually check.
Your problems are therefore going to be:
- What facts do you want checked?
- Does your tool include a check for that fact?
- Does your tool check that fact in the way you’re hoping it checks that fact?
And if you can find out those things then you’ll have your answer.
You may actually not want to examine coverage. Coverage is a weird concept, because software isn’t a thing with surfaces that you can cover. Testing software is infinite, and every possible check you can run can be run ever so slightly differently to make it a different check. You don’t have coverage of software, only coverage of models of software, so you’d need to decide what that model is, what it looks like and what it contains before you can pin down what coverage means to you.
Code coverage is one kind of model, and you might make one particular model of the code against which you might check coverage. However this is only one model of many. Checking code doesn’t examine all risk and checking risk doesn’t examine all code.
You can cover function, and you can cover data, but functions and data interact. You can cover requirements, but those are only written requirements in a document and don’t include the requirements not written in the document. Plus the descriptions of requirements in the documents can’t always be written out in code - or leave important gaps between the requirement and the check.
So what can you do? Well, one way is to go through your automation tool and describe the facts it checks, and why it checks each fact. That’ll give you an understanding of what the tool’s doing, and you can decide if that’s sufficient coverage for whatever model you have in mind. That is quite resource-consuming.
You could take shortcuts by, for example, categorising these fact checks by function or product area. If you have no fact checks for a particular area you can then ask yourself if you should or not. If you have many fact checks for a particular area you can ask why. Many fact checks might be the result of a static bit of functionality that doesn’t change, so maintenance on the checks is cheap. It could be that it lends itself to specificity, such as functionality that has numeric processing or more formal logic - stuff that has to work in a specific way, like trading or financial software. It might be that the team that works on it writes a lot of checks.
This will give you clues you can follow to ask more questions and that will more likely show you where effort is spent.
And most of all, consider risk in your product. This could be an excellent way to start a model against which you consider coverage. Automation is another expensive software project on top of an existing expensive software project, so consider what facts are important enough to be included, and then look at the automation tool to see if those facts are checked, and in the right way.
I understand the irritation with understanding the contents of an automation project. Even comparing the description with what it actually does is a minefield. I understand that my reply isn’t convenient, but it’s as close to realistic as I can get. Hope some of it helps, anyway.