I was wondering how you guys measure the success of your automated test scripts.
In my current role weâre measuring â% of test cases automatedâ and itâs in our goals to have 80% of our API test cases automated (mixture of checking data integrity and HTTP response codes are correct).
Other ways I can think is:
Anyone got any other ideas?
Time saved is probably one of the main metrics used for justifying automation efforts because it is easy to turn that into âmoney savedâ. You should adjust the time based on maintenance as Gabe suggested in the fragility metric. I prefer to stay away from â# of testsâ and âdefects foundâ metrics because they are deceiving. Having a large number of tests doesnât mean that the right things are being tested (similar issue with code coverage). âDefects foundâ can be even more discouraging because if there is a large number, someone looks bad (devs for bad code or automator for false negatives) but if there are none, you donât know if that is because there werenât any or if the tests donât address the needed scenario.
If the tests directly represent business paths, I would consider the number completed to be more relevant. The difference is that the tests can be mapped directly to the business in those cases. For example, 80% of the common paths through the feature are covered by automation along with 5 high risk edge cases.
In general, I suggest avoiding metrics for the sake of metrics and try to find something that is an indicator of why you are automating and itâs importance to the businessâŚ
Thank you, Ali. This is a question I realise I hadnât pondered enough on.
Test Automation can take many different forms. One form is to prepare test data another is to put the system into a certain state or you could have your automation scan for broken links. These can be immensely useful and their worth is rather easy to measure.
An automated regression set (on any level) is something different though.
This usually takes a huge amount of time to decide on strategy, implementation and maintenance so the cost vs. benefit exercise is quite important to do multiple times during itâs lifecycle. And readjust where necessary.
At first, measuring the success is as binary as the results of the automation.
Red/fail: a problem is found. The automation successfully prevented an issue to move over to the next step.
â> Measure: Number of bugs found.
Green/pass: The application passed with no problems found. You now have some added confidence in the stability of the product.
â> Measure: Number of checks passed.
However, this completely ignores the hazard of False Positives (there is a problem, but wasnât picked up) and False Alarms (a problem was found, but it turned out not to be a problem).
Though, when we think further about both measures: they both actually give us more of an indication of how the stability our development/environment/⌠is NOT how good our automation is. The goal of the automation is not to find problems or to give confidence. It is to be faster and more reliable at menial tasks. The aforementioned are a by-product and should not be the main focus. Treating them as such may be counter productive at the least.
To know how successful the automation is one should consider coverage, reliability and mitigated risk.
Add to that the time invested in creation, maintaining and running the scripts (and chasing false alarms/false positives) and youâll have a rather useful âmeasureâ of the success of your regression suite.
None of these parameters can be expressed in meaningful numbers though, except for time.
(this completely negates the psychological effects that âhaving a regression setâ has on a team & project.)
Therefore, the repeating question of âshould we invest/keep investing in a rigorous automation regression set?â is often very hard to answer. Many different questions should go along, such as âHow many times do we build or release to production?â or âWhat is the root cause of most of our issues?â or âWhat kind of risk should we tackle?â.
Iâm often very grateful of having a good automated regression set and love discussing the strategy it should be part of.
However, Iâd be very wary of any measure of success even if it looks incredibly crafty and sensible.
Hope that helps give you some insight.
At least, writing it out helped me get a clearer idea. 
I have around 8 Years of experience, Worked on different platform(Mobile/Web; Windows/Mac) and SAAS APP automation.
Following below are benefit of automation getting success:
Fine, give all the good answers first, see if I care.
Ali, Iâd strongly recommend pondering the details in Berenâs already excellent reply, especially those about mitigated risk. My automated test scripts (âautomatic check scriptsâ as I call them) serve the purpose of checking the probable validity of a fact because it serves some part of my test mission. If I know why Iâm running the script and I know that the script reasonably serves its purpose to help me find out something about the product under test that is valuable to my mission within a reasonable cost then I consider it successful. The problem is the nature of those scripts changes depending on what they are and why youâre running them. Some of them I write in half an hour, run them for a test session, then pretty much throw them all away. Some of them are in a CI build to help reduce regression risk - a different purpose for a different mission that needs different reasons to justify its cost. So you need to know what those cases are trying to achieve in order to establish their success.
A useful heuristic here is to consider how measurements of automation fail to describe the purpose of your automation. You could measure the number of lines of code, for example, which would be never too far away from being pointless. The ânumber of bugs foundâ hides the nature of those bugs. The number of times a script does not find a problem is equal to the number of times a regression (that would have been detected by that script) has not occurred. The time saved by running scripts is balanced by the lack of more powerful testing that makes better use of the unique powers of inquisitive humans.
Interesting point to be made that the way to maximise time saved would be to do no testing at all, yielding a maximum money saved. If you mean time saved instead of humans doing the work then I think itâs really difficult to translate it into money saved - thereâs all the costs of training, running and maintaining a code project, not to mention that automation doesnât replace human testing so you have the costs of the problems not found by checks that would have been discovered by a person. Youâd have to quantify the yet-unknown abstraction failures.
So I think the avoiding metrics for the sake of metrics point you made is extremely important. Iâm also very much with you on the âwhy you are automatingâ point here, and I think itâs absolutely key - itâs hard to translate the cost-efficient but suitably thorough search for problems in an infinite space into quantifiable terms. I think itâs easier to present the case that your automation serves a purpose that justifies its cost.
âTime savedâ was intended as a reference to the different methods by which the tests could be run. The assumption is that if the discussion is happening, not testing is already off the table. As for estimating monetary value, the practice I have seen used is quite simple: calculate the difference between the automated and manual runs, multiply that by an hourly dollar amount considered to be the approximate cost for a manual tester. That value is then reported as a savings on manual testing costs. When viewing testing as a whole, the same process is performed for the creation and maintenance of the automation and that amount is deducted from the overall savings. If you are viewing the short term, manual testing will always appear cheaper. When you extend the time-frame viewed you will see that in the long term automation will provide better value. If being unable to quantify the unknown was a valid reason for not creating a metric or estimate then we wouldnât be having this discussion at all.
Hi Ali,
Wow this is a big one 
One thought I have on this this is to ask yourself the question of why are creating automated scripts? What is their purpose? The metric you mentioned (% test cases automated) is metric that doesnât seem to look at their value. It measures how successful you are at creating automated test cases, but it doesnât measure how much value they are adding to your company.
I will say that measuring stuff like this is extremely hard (and sometimes dangerous) as metrics are slippery things. We usually canât actually measure the things we really care about (improved quality for example) and so we measure things that approximate them.
Some things that I have used and/or am experimenting with using are:
The assumption is that if the discussion is happening, not testing is already off the table
Yes, my point was a reductio ad absurdum to show that measuring success by reducing the time taken means that the pinnacle of success would be to do no testing. Therefore success cannot be a product only of time reduction.
That is a simple practice, but I question its utility, because automated checks and manual testing (testing without large check suites) are not fungible. Measuring time is useful when we look at automated check suites as a way to support a wider testing effort, because we want to understand the costs of running repeated checks, but not as a way to replace or compare one with the other. A coded check is not the same as a human doing testing attempting a check which is not the same as a human doing testing in general. You can measure the amount of time taken in each case, but it doesnât tell us anything about what weâve learned about the product, what problems we find, what risks we uncovered, how we told our story, how we informed our test clients, how it helped them to make better decisions and the perceived quality of the product. It doesnât take into account how each thing serves our test mission. All we know is that it might take less time - so weâve saved money (maybe) but potentially lost value. Time estimation only shows us which is quicker, not which is better or more successful⌠not that it really matters because you canât automate testing anyway. Thatâs what I was trying to get at there.
Well to measure automation success we can take into account different benefits of using automation, apart from all those the two biggest factors which I consider for a automation to be successful are
One hand-wavy metric I use to measure success is whether automation allows me to do things that would be infeasible to do manually. Itâs not the usual quantitative metric that would look good in a status report, but it definitely scores some points if you can demonstrate it.
For example, my team was able to measure the runtime performance cost of different debug info formats for the Linux kernel. It required running a slew of benchmarks and tests across a wide variety of hardware and then analysing the results. If we had been unable to automate all of that, we never would have even attempted it.
Interesting topic. To me the measure of success of test automation is how often can you release? How long does it take for a piece of code to go from check in to production.
If we talk minutes then it is a success
The SUT appears to be some form of web based system, so I would ask what platforms and operating system versions are used/tested?
Personally I would never automate to find defect, I would automate only for regression testing and for data injection for performance purposes.
In the case of web services at the user experience side you need eyes on AKA a tester checking that all is correct, you can then move on to automate this to cover all platforms and operating systems/browsers.
Great topic!
To me, successful automation is when everyone in the team has confidence that requirements of the product are fulfilled. When developers get fast feedback after their valuable code changes, when regression doesnât reach environments where real testing happens, when testing is not about verifying that something that used to work still works. Simply put, when everyone in the team spends their time only on what is valuable and requires their skills and knowledge.
I agree that metrics can be a slippery slope. What is important to stress here is that once you start providing certain metrics, even if they are not the best, but your stakeholders/managers may cling to them, so itâs better to avoid metrics rather than provide something that becomes a burden and does not really reflect the situation. And all of what you provide, you should find valuable yourself.
What is automation success for you?
The usual case is that manual tasks were replaced. Then, time gets into picture. For teams working in Continuous Delivery this would lead to question: how long does it take to run the tests before releasing? We need fast feedback in agile, so would be great to do frequent deployments and get answers immediately.
If all the tests passed - this does not make me think that automation is successful. What I have heard of before is that if the test is passing all the time - itâs rather useless. If it passes for 5 years each time - do we really need it? Is it really passing or it does not work anymore? We forget. Test which is always green is nice to look at, may add to nice numbers, but what does it actually test and do we need it? That should be a question. Some extra logging with test results could be useful and then the analysis of logs.
Automationâs value is in being a helper for repetitive tasks. There was a saying somewhere that if you need to do something more than once in testing - you should automate it. I believe that is the biggest success.
Also, looking at the situationâs description in the original ticket, what could be useful is if the tests are created on the right levels. Is there a test pyramid? We donât need exact numbers, but we need balanced ratios and this could help with fast feedback for CD.
In summary, metrics are a tricky topic and they may not answer automation success, but a few which I would consider:
Hi Ali,
This is a good question when doing automation testing.
Below are some criterials to measure automation success based on my experience:
Hi all,
This is a regular question and always be asked by the client frequently
Below are some ideas to measure automation success based on my experience:
We should analyze, estimate the automation difficult thing and the risk of your project. After that, you will have the number of test cases can be covered as much as possible by automation. I worked on some projects that the coverage was about 50%, and the others should be 90% or 100%. But all are successful. Itâs depending on the project that you implemented automation for.
The framework should be clear and has full support like log/result/reports/debugs/Data-Driven. Currently, the framework should be clean code, low cost of maintenance, easy to understand and maintenance by the others and should be able to integrate with CICD tools.
Track the time when running the regression test by automation to make sure the automation framework decrease the time compare with the manual test. If the time is not decreased so much or be increased or the same, so we should consider doing automation or not, the framework and scripts that you implemented are not successful.
For the framework team, should design the framework so that it is cleaner and easier to be used by the manual QA or scripting QA.
The scripter should be trained the clean code and written the stable scripts. We must run automation regression test weekly or daily. So the stable scripts will help us decrease the time to invest, debug and fix the scripts.
When I think immediately after reading this question, I say I would measure it this way :
a. Value the tests bring.
b. Reliability of test results.
c. Ease of use.
We may check the automation success on several parameters if we talk about different software testing companies approach. Most of the projects are now being tested by automation only as it saves lots of time and efforts.
Hope this is helpful for you.