So, I’ve seen a lot of companies preach ‘we do E2E testing on the UI’ and it works.
BUTTT they never calculate everything in their setup cost, maintenance cost, cost of people working on it, CI/CD cost etc
There is always 1 guy 24/7 fixing ui tests due to ‘flakiness’… so yea I’m wondering if you have ever made the Cost Comparison and if there was a ROI on your UI Automated tests? Because I’ve yet to see it :x
Looking for stories & some calculations positive & negative XD
Unfortunately a lot of the sales pitches are based on the fallacy that the starting point is having lots of people rotely going through a mountain of test scripts.
So its a bit like envisaging a starting point that if human one did what a machine can do fairly efficiently and then he repeated that across lots of devices, browsers and OS versions that would be a lifetime of wasted low value effort.
Anyone using the savings of human not doing that mundane repetitive low value activity as part of their automation ROI are going to be able to propose fantastic returns.
Then again its a fairly absurd model to use, its likely best to avoid any company or tool that has a human one type effort or cost saving activity in their ROI.
The ROI really should be more about the value it can add rather than what it can save.
To me it’s usually about what it can catch, does at catch important issues quickly and efficiently that may be missed otherwise or not found till later that would have a cost. What’s the value of what it is finding.
The ROI on this is fairly hard to calculate and for the most part I’m usually in favour of keeping UI automation fairly light focusing on key user flows that have been known to regress at times in the past, they are key and if those flows are actually broken the cost is high. The even harder part with this ROI is that if your developers are doing a great job your tests may even have very low ROI which though can be great. The rapid feedback loop can be a good ROI as well, the value for your developers to build at optimal pace confidently is good too but again hard to measure
When its kept fairly light, you often will not need to spend time calculating an ROI, managers, customers etc like automation even if they do not understand the value, they often trust it is of value, which it is.
Because its hard to calculate though you may find sellers opting for the fairly lazy and easy cost, effort, time saving of human one in their ROI even though its pretty much a fallacy.
The biggest positive ROI for E2E tests for me were on a massively technical debt ridden nightmare. I’d argue it saved the project.
I spent weeks building a test harness and ironing out the flakiness, but it was worth it.
Other types of testing (e.g. unit) fell flat on their face because they straddled service boundaries that badly needed refactoring. Once they were refactored, the test became useless.
The lowest ROI for me was where I tried implementing it on a project where the login code itself was flaky as hell. That didnt make it impossible to test, but it made the average test take > 45 seconds, because the test had to keep trying to log in. I didnt stay long enough to fix the login code.
I dont think the ROI is necessarily possible to calculate up front and is highly contingent upon circumstances. I think you just have to build a few tests and see what happens.
I argue that I want to see few but important business-driven UI test workflows, because at least they will be controlled. At the end, when the last change has to be incorporated into the code to be delivered, you can at least be sure that the most important things still work. Testing the variance via GUI’s really makes no sense. Unit or API tests are much quicker and, above all, even better than UI tests. It is also important to me that the test executions are carried out regardless of the tester’s state, i.e. the prior knowledge does not have to be there, the tester does not have to be in a good mood and test tools do things fairly accurately.
The important thing in all of this is that the maintenance work is low, the operators understand their tool, enjoy writing and managing tests and are not just the “hero” of test automation.
What would be included in such a calculation?
And what would be the criteria for something to be a ‘return’?
I’m thinking about cost to build/setup/maintain the setup. (Cost of the people)
Cost of tool, pipeline etc … and all the things I don’t recall now 
So what is included? I don’t 100% know and I’m looking for other people who did this.
We are talking about RIO here so the investment cost should be smaller then the return cost.
So the cost of letting UI E2E run in the pipeline would take for example 20 minutes. Compared to Doing this manually 2 hours.
If you run it every day for a year you’d win X hours/days - setup/maintenance cost/pipelinecost etc etc…
Something like this? I’m looking for an answer too 
@kristof I’d really advise against any automation ROI to include any reference to hands-on testing.
They are not the same.
If you have people doing rote script execution by hand then revisit the value of that separately, why are they doing it, what value does it bring, how much waste does it carry.
It tends to be a false premise for automation albeit a common one that is often used to sell automation, its an easy ROI as per my earlier comment but that does not make it useful. It could take man-years for a human to do what an automation suite can do in minutes but they’d never actually do it so its not real saving.
The ROI of stopping doing hands on scripted testing and doing better testing can make sense.
For UI automation though, consider it separately. Your decision to leverage automation follows the same consideration as any other tool; what problem are you trying to solve, what are your goals and the expected value you want it to provide. Are there better ways to achieve those goals.
What’s the value of the rapid feedback loops to developers, to have confirmation that you core user flows are still running, to broaden coverage across devices/OS/version variations. Maybe even the morale, boredom and disengagement aspect of people doing repetitive mundane tasks could feature in.
There will likely be some level of cost and effort savings but I still recommend removing that from consideration and look at the value it can bring.
What important things has your suite caught that would have been missed or delayed otherwise and what was the value of that. This can be a tough question but it is important.
I wish it were this easy in the IT industry and we can throw millions of dollars towards products
If I automate several important tests but it takes way longer to develop then test it manual 1000x, then I rather have it tested manually.
If I now say we have to work in style X or Y or buy a new tool but it has no return then why do it? (automation, agile, new fancy tool, any kind of work) My point it, I can see the value in it… BUT it also has to be reasonable to do and cost affective.
Compare it going to the bakkery and buying a sandwich with 1€ of meat or buying a sandwich with 100€ of meat. The expansive one will taste good and and it will have a lot of value for my belly BUT it’s a 100€ and that might not be worth it for my wallet to do this every single day. Because in the end in 1 year, we have eaten for either 365€’s or 36500€’s and that’s a big difference, even though 1 day doesn’t hurt.
Then the question becomes what value do you see in it?
Try another question, what value do you see in testing it manually?
Do you have an ROI for that, what is that ROI based on, what value does it bring?
What you tend to see from automation suppliers is that they do not even consider the ROI of testing in the first place and their ROI of automation is then based purely on replacing something, maybe doing it faster and more efficiently than something they have no idea of the value of the thing they are replacing.
I’m suggesting this is a big red flag.
Look at both your hands on testing and the leveraging from automation tools fairly independently, they have very different goals and values, if one can be interchanged with the other its likely something is wrong in the first place.
When the focus of the ROI is on the value it adds as opposed to saving from a false premise you will likely get a clearer idea of its real value.
Lets take your sandwich consideration. The sandwich has value to you when you eat it manually but it may take your lunch hour to get it and eat it, thats a time cost.
Now lets automate that and the machine can eat it in two seconds, the ROI saving is almost an hour but the value is gone because they are not the same.
But you might decide to automate the delivery of the sandwich saving 20 mins, that might be worth it and then we consider the cost which could be financial and weigh that up.
The point is to consider the core value of eating the sandwich first then separately consider the value of the tool otherwise you will have a great ROI of an hour a day at a cost when you could just not have the sandwich at all and make the same saving without cost, yet also without value.
I don’t think you understand the original question because this is what this post is about, looking for a way to calculate the RIO?
I can see E2E being good and remove the repetitive work but I’m just looking for a way to calculate or see if there is an RIO or not.
I’m not pro I’m not con. I’m just trying to see the cost of value 1 or value 2.
Value 1 being manual testing and Value 2 being UI automation.
Exactly my point, in your example your machine eats it in 2 seconds and the ROI is saving almost an hour BUT the time to create that machine isn’t in your calculation, neither is the maintenance and uptime. So if building your machine took 1 month. It takes many sandwiches to eat before you have ROI and that’s what I’m looking for.
This is exactly what I mean, you are assuming there is an RIO but I’m actually “assuming” there isn’t and that’s the point I’m trying to “prove/work out” for UI automation.
I do understand your point of ‘value’ and I agree but it’s irrelevant at this point imho. Because there is value in UI automation for me BUT not at the cost of not having a RIO. That’s my “value” that I’m trying to calculate.
Okay, I might have gone off at a tangent.
But I am trying to understand the value you are seeing in the UI automation?
Now the costs would be part of that but I have not picked up on how you are defining value for these automated UI tests and that would normally be the starting point before looking at the costs.
Without mentioning manual test effort, costs or savings, can you define what value you will gain from automation at the UI layer?
Just as an FYI.
I’m currently working on a Flutter Patrol suite, we opted to do it as R&D initially because the value is fairly vague, qualitative rather than quantitative initially.
As you mention we know its of value but find it hard to qualify that value, even to the extent it may take six months of running these suites before we can determine its real value. Does it catch important things quickly that we would have missed otherwise or the delay in finding them in other ways would have carried extra cost. Does it allow the developers to have confidence to develop at a faster pace than without it?
One we have done the R&D project and can get numbers we may then be able to extrapolate that value to other projects and propose a more accurate ROI on those.
This initial effort has been higher than expected, mostly mobile and CI config setup and maintenance issue and as a one off endeavor I doubt the ROI would be justified. On future projects though it maybe as we’ve gone through a lot of the hard work and costs will be lower on new similar projects as a result.
I could though sell the ROI very easily by suggesting it does one man year of manual effort every day, it does but its not real so I avoid those type of ROI and only look at the potential value it adds.
I am struggling with the same challenge I believe you are. Taking a cost initially to get a better view can be worth it.
Let’s say the usual
I guess these are all always related to effort, costs or savings somehow.
Exactly! I’m also having a hard time justifying UI Automation at all. Hence I was wondering here if somebody had made some calculation towards it. Because with the current “effort, costs, maintenance” I cannot justify it at all.
An Automated UI test has curious benefits, for example it will be testing the business logic layer in your app, where-as our white-box-tests and component-tests will not cover some of the validation and business code that sites in the UI layer just by virtue of how those UI’s are often composed. So an E2E test of the Ui covers the boundary cases that the UI presents, not the boundary cases that the back-end exposes. That is valuable in itself.
Secondly, when you automate the UI, you gain ability to run more experiments and more realistic stress tests over the client deliverable’s performance. Yeah, it is terribly hard work, but if you do not aim for specific goals in UI E2E, but fall into the trap of testing every single page in your app, it can become soul destroying, something that has little business value.
/edit
Thirdly , if your UI automation is language agnostic (meaning it uses element IDs to navigate not the text) you can use the automated test to drive the app in all supported languages and then verify that you don’t have invisible text on any translations. I have used this successfully many years ago on an embedded LCD display app, so I know it’s a valid thing to test for.
One interesting thing about ROI is that you might not always expect money saved in return.
I heard from a manager once that half a million euros in automation costs for an E2E product is fine with him. The gain for them was the confidence given for the product releases. No more thought would be put into any other calculations.
Another manager was referring to the expense/revenue to the business value. A couple of attempted automation projects(pushed by other managers) costing 150k euros were enough for them to say no more. ‘If it’s not something stable, and quick, it’s not worth it. The risk and impact on the business of any issue we might miss is low with the current testing. We could get a full-time senior developer who can increase our product revenue by x% in the next year by implementing the Feature that we’ve already prepared and it’s in the backlog.’
When there’s a consultancy project, from the consultancy perspective, the automation project ROI is its acceptance by the company buying the service.
In a request for proposals from a consulting company I’ve seen the following calculations for the ROI:
The product under scope: a transportation management system, total cost of implementation was estimated at ~15 million
Automation target:
We have to test our client on the UI level. It was built without proper architecture and just organically grew. Therefore the API input gets changed and supplemented by the client making API tests useless.
Our regression test automation project began at a point where the client was mostly finished and little change expected. We don’t have much issues with flaky tests and finished much earlier than expected. It was definitely worth the effort. It’s not just saving time in regression testing, but also a good description of behavior as the original requirements were lost if they ever properly existed.
That, kind of timescale is a bit insane in todays world where app frameworks evolve so fast. You might want to look at using your investment in automation as an “oracle” to help you port your product between frameworks, as well as as just a plain testing tool. But I have to say, even hardware does not last that long, nor to employees.
UI automation is possible, however your elements must have given names and should not change when there is a new build.
for windows UI , you need to program or web you can use selenium . However, it’s still checking to assert that nothing is changed when testing a new build.
there are companies who test their APPs on a test bench by sending a instruction to control the GUI.
Experimental/exploratory UI is often more useful, the human eye sees more. For example if you want to test a website for colorblindness, or observe strange web site behavior.
If automation is done incorrectly you indeed can have flaky tests
I’m going to share a UI bug, that, for me never gets old.
The devs had spent months re-implementing the console of the enterprise back-end tool. It was a framework change but the first version was a pixel perfect copy of the old console, it literally looked and worked identically except.
When you launched the beautiful GUI console you could click the mouse to choose which area of the system you wanted to go to, it was very graphical, all of our UI automated tests were passing, and then the first customer defect report came in. I had a look at our automation script.
...
def open_config()
launch_process()
click_configure()
click_configure()
...
And that was our bug, the automation tests had long ago found that the app was not setting focus to the inner frame in order for the new framework to start accepting mouse or keyboard input events. Every auto test was clicking on the first ever button in the console twice, the first time to set focus to the inner frame, and then once to actually tap the button. Manual testers where opening the app, and clicking twice, because they were all assuming that the app was merely not the “topmost” and in-focus window yet. To this day, I take automation testing far more seriously, because none of our manual testers found this bug, and the script jockeys just worked around it.
The developers made a code change to wire up the input event handler at the right point and we shipped the fix in a patch the next week. Happy ending, but it was so so funny, because neither manual nor automation testers found the bug. I’m sure I have many more, but this is one example of why automated tests find bugs that manual testing often will not.
Its interesting I thought there might be a few more responses to this, maybe from a few members directly involved with automation tools.
Maybe without the focus on the cost/end effort savings this is actually fairly hard to to do ROI’s.
I still see value in UI automation, the rapid feedback loops, broader coverage, quick and fast coverage of some the e2e elements that other layers might miss. Health check benefits that indicate ready for deeper testing. Combinations, timing and maybe complexity that machines are better at. Increased regression risk coverage. The idea it can speed up development.
I still like the question, what did it find that you may have missed otherwise or the delay in finding would have slowed down development. You may not know this part for six months though, but if you can demonstrate it on one project its likely fair to add this to ROI on the next similar project unless of course you instead address the root cause of the issues.
So getting buy in to the ideas of the above have value but still very challenging to put a quantitative rather than qualitive ROI on it.
Wondering if the tool suppliers can do this better without talking about cost or effort savings?