I was reading this interesting post on a testing Slack:
The company wants to hire an automation engineer, the lead tester wants a tester instead.
What they are doing for regression is using a questionnaire and then also having a meeting with a tester, dev, and architect each sprint to narrow down the potential regressions areas to be covered. Using this strategy, in a 2-week sprint, a tester allocates 12 minutes to specifically check the regression for a dev task.
Per sprint, there are about 50 tasks. A total of 8 hours are dedicated to regression checks.
The approach is exploratory, testing around the area with risks, having change knowledge, domain knowledge, and some test artifacts.
The testing team then spends the rest of their time testing the new implementations.
An automation engineer would cost about 80 hours per sprint. They will be limited in the coverage compared to explorations, which can be a disadvantage due to missing potential problems.
Does anyone want to calculate in how many years an automation engineer building a regression checking framework would manage to come up on top?
Well we haven’t done an actual ROI in terms of cost of establishing UI test automation vs it benefit in terms of reduction in man hours of regression + bug leakage reduction to higher environment + early bug detection with CI/CD regression (only on impacted area(in micros service architecture)).
But its definitely has positive return when its planned thoughtfully.
Plan your resources and effort carefully.
Do prioritisation and selection of regression suit carefully(High priority E2E test that will cover max validation).
Calculate how much time it will save for your functional testing team over an year.
Now you have both estimation in your hand , and it will help you a lot in deciding your next step.
What i experience is starting small have the max ROI , you start with only sanity/smoke test automation, Integrate them in CI/CD with each QA build and it will save you 4-5 hours a week.
Then divide your regression in module and prioritise them based on business criticality.
@yogendra_porwal I recommend that you look at the ROI without looking at cost or effort savings from other activities and focus on the other things you mention.
That saving is often not real or if it is real then its likely the team have been doing activities that favour machines rather than testing to human strengths.
I went a bit more into detail on this in my other comments, they may come across as harsh comments in some cases, but still may be useful to consider looking at things a bit differently than those with a vested interest in showing easy ROI’s for automation.
Hi @andrewkelly2555
I’ll look at your document. But still i differ with comment “if it is real then its likely the team have been doing activities that favour machines rather than testing to human strengths.”
Test automation actually help to free up time for your testing team so they can focus on actual testing, instead of doing repetitive task(regression) at the end of every release.
I have seen that ROI not in terms of cost and effort but in terms of better quality of product.
For eg. I have been part of team as a single QA for the application and faced below problem.
Always have less time to test feature as dev-QA ratio is not balanced.
Increased regression suit release after release.
Pressure from stack holder to release on time. (Even when regression is not completed).
Test automation did helped me to free a lot of time from:
At least 4-5 hours/week saved from Smoke test after every QA build
Couple of days per release from Regression before signing off.
Its not like i had 100% coverage (its impossible). But i focused on automating only E2E UI test that can cover most of our UI validations (apx 70% i would say).
Most of our business logics are covered in API test automation.
It seems like this waste has be created internally and then the saving against that waste is used to justify the automation.
Consider two scenarios.
A - You don’t have any decent regression coverage, you look to automate some of this regression risk coverage. You are not stopping or replacing anything. In simple terms say the cost was 5 and the value was 10.
B - You put in place some human regression risk coverage and then you look to replace it with automation. The cost of the human side was 2 “weekly” and value 1.
Now you replace with automation and also expand to be the same as A. Cost 5 value 10. On the face of it looks like the better deal. You have saved a cost of 8 a month freeing up those humans and also got the value of A cost 5, value 10. It seems like a magical cost of -3 for value 10 . This is what automation sales pitches often do.
In reality though B has cost you all the wasted time you did the human element say 3 months so 24 plus the automation 5 for the actual value 10.
In both cases though you end up the exact same place though, cost 5 value 10.
Using model B, hides the reality and includes past mistakes to justify a false ROI.
