To start a new year, I have been sorting an argument where a project manager pitted two testers against each other without their knowledge on estimating efforts to fixing test automation that had not been maintained for a while. Two people guessing how long a thing might take, for themselves or for others is bound to deliver two essentially different guesses, and the biggest point of conversation ended up being on where people put the work they know will take time but can’t identify in their work breakdown structure.
This lead me here to ask on how people these days approach the questions around “estimate how long it takes to fix these 200 tests where 100 are currently failing" or “how long to automate when we have 1500 scenarios and it took 40 people a full month to test them manually”.
I tend to go with: “pairs that share progress every two weeks, forecast from there” but how others go about this?
I had +6 years ago such a project. 3 years nobody fixed the process test automation and most of the tests failed. The quality of delivery, not really a surprise, went down in that time and the project asked me if I could help. My estimation: Each failed test need 1/2 day because I was involved in creating the test cases and knew their complexity. Some could be fixed faster, some would take longer I thought. Also, if global routines will be fixed, some test cases will be fixed in parallel.
My estimation matched. After 3 month I was done and the tests were green again or the bugs in the SUT has been fixed or the process changes now being relected in the test case.
Same estimation we did +15 years ago as we had trouble in dependency of test run sequences of a project. We estimated also 1/2 day for a test case because the test automation engineer was familar with the SUT and the test cases. Also there: It machted nearly our estimation.
For me I made a rule:
If a test automation engineer is knowing the SUT + test framework and should fix a number of test cases, just take the halv number as days for estimation.
I have another project where I look at this from the other way around. In 2 months of effort, 2 test cases have been created. There was definitely the “not familiar with SUT” in play, but also a newbie in test automation.
I find I have a lot of clients who seem to be using your estimation mechanism, like right now I had one where they say they have 20 scenarios and willing to give two weeks to automate those.
If the testers don’t have any domain knowledge, if the test validates the correct scenario, you need more time. Perhaps do a spike, let the testers do some exploratory testing (I prevent to use manual testing), and get familiar with the product and required tools. Perhaps that test framework has a deeper problem in the libraries used. Sometimes (timing) issues can be resolved there.
This is a difficult one as it depends on the organisation and its workflow - so . The way you state this problem, its as if the organisation sees this work as a “project” that needs estimating planning ands tracking until its completed.
In my world, I wouldn’t estimate automation maintenance as a project. Because its not a project (unless I’m misunderstanding and you are delivering to a customer the test framework as “the product”). The fact is the framework wasn’t maintained, thats the problem. Test maintenance should be happening all the time so we would raise tickets for a sprint as to what maintenance we want to target in each sprint.
But the first step would be to understand, how it got into this mess in the first place? Was the product shelved for a long time and then came back or were the tests high maintenance?
If the latter, decide if its better to refactor the framework or start again. Been there!
If the former (and your happy the framework should be low maintenance day to day), than we would probably document an approach as to how we catch up in the most efficient way. That would include assessing if the existing tests are worth the effort, if functional priorities have changed, looking at the test approach as a whole so we prioritise what tests we get working first - balancing the effort with tactical exploratory testing as development continues. Track that automated test coverage to see the rate its increasing and if releases are needed before this maintenance is finished, you’ll have the highest priority tests automated and what you haven’t automated you have to cover with exploratory. That coverage gets better with each release.
For me, we’re not delivering frameworks, we’re delivering products. So we wouldn’t have a project manager tracking where we were with the framework, thats an engineering problem. They only care the product is delivered with good quality and on time. How we do that, isn’t their problem. However, engineering would track the effectiveness, coverage and efficiency of the framework.
Maybe I’m reading into this a bit too much but there is are a couple of things to unpack in the question…
My first question is why is a PM asking testers to fix the test automation suite in the first place? Red flag.
Second red flag, the PM is playing games by secretly pitting testers estimations against each other - for what gain? So they can point out the discrepancies of two completely different opinions / approaches? So they can pick the ‘quickest’ estimation? Doesn’t sound like a helpful PM to me, sounds like they’re wasting peoples time and avoiding a proper strategy to solving the challenges.
Questions like “estimate how long it takes to fix these 200 tests where 100 are currently failing" are not realistic to ask. It’s a problem statement, so you’re better off stating the problem clearly; ‘we have 100 tests failing in our test suite of 200, what’s the approach to solving these?’ is better framing, because it’s at least open to ideas, rather than ‘these need fixing, how long?’
“how long to automate when we have 1500 scenarios and it took 40 people a full month to test them manually” - another unrealistic ask, and is again more of a problem statement. I.e ‘We need to reduce the pain of manually testing these 1500 scenarios, lets look at what can be automated and create a plan to move forward’
So to answer your question -An approach to the problems… Well, if a test suite has 100 failing tests then there is a clear lack of responsibility within the team that needs addressing. How did it come to 100 failing tests? Why is no one taking responsibility for it now? Current skill level of the team? Was the framework created by a third party? Is it too complicated for what it does? etc.
Then it would require a culture shift to make the automation failures part of people’s role, so they’re taken seriously - as well as the practical task of solving the 100 failures, which as you suggested can only be done in a progressive, by-weekly update sort of way.
There might be 100 things to fix, they’re might be 1, until you dig into it there is no way of knowing this off the bat.
These are many different organizations. I was sharing bad examples not to seek answers to how exactly I would estimate this but on when the world is this messy, what are *your approaches to chunking the budget piece you need to set up reasonable expectations. *
My way is usually having people join in pairs in 2 week chunks rather than ever giving an estimate. Then again, I am measuring one project where 3 months of effort has produced 3 fairly simple tests, and wondering how people end up talking about the differences of how long it takes depending on your background.
If the test effort involves 40 people for an entire month purely for execution of 1500 test cases then I’d assume a high level of complexity or some ancient and inefficient systems. Either way, I wouldn’t touch an estimate without a small case study and a guarantee that resources are adequate. For some reason, companies seem to think that QA resources are completely interchangeable whereas in the real world a variety of skills are on offer with different pros and cons for the task in hand. For example - making a dent in 1500 new automated tests might benefit domain knowledge over pure programming skill
It all depends on the context of the domain, product, process, people skills, and scope.
Generally, it is hard to predict how much time will be needed to fix or investigate the issue.
Sometimes it took an hour, and a common problem fixes dozens (or hundreds) of tests. Sometimes it took a day or a few days to find a root cause. The more technical and product knowledge the tester has, the quicker they can find a problem. To make estimations for fixes more reliable, we need to track each root cause and make this data usable and searchable. Do not store this info only in the mind of only senior person.
Adding new automated tests depends on the available toolset, technologies, and … the current automation solution. Sometimes the solution is poorly designed, so adding even a simple test for a trivial feature will lead to weeks of refactoring.
I have had to start answering this question for myself this week, because I finally hit a point where I can say the backlog is hitting steady-state. (Things to test no longer get added faster than I can take them off.) Migrating a team from a manual testing and unit testing only to having a CI/CD with component tests and E2E tests that actually target platforms, takes longer than just 2 months.
I had 2 job interviews 9 months ago, in one I estimated around 2 months to build a fledgeling test automation system, for a simple IOT device with some web services. In another interview I doubled my estimate, because the target was a bit bigger too. Feedback on 1st was that I was too optimistic, and I did not get the job there. So, 8-9 months later here I am solo and starting to fling new test cases into a slightly creaky TeamCity test machine farm. I was more realistic in the job interview this time. Which probably clocks in at around 1400 person-hours. A fair bit of that was domain learning as I was and still am fresh to the particular embedded domain in question. At the moment I’m sitting at around 15 hours per test case, that needs to come down, and is finally happening this month as I have used 8 months to build up my kit of tools and product knowledge.
Whatever people tell you, double it. I have never seen my new testing system as trying to “make a dent”, just one suite of component tests flushed out a fair few “inconsequential” or non-functional bugs for example. About a dozen bugs, in just a day or two. We decided to not fix many of them but to rather move forward to the next component and other scenarios instead of trying to finish off that suite and go deeper. Everyone’s context will differ but my context is to only write tests for bugs that customers report today. I’m taking a shortcut I know: namely Not to test stuff that is unlikely to break because it might be static/legacy code. But 1400 hours from scratch I hope, is normal. (Linux and Windows.)
