Hi. I know many teams out there have weeks of regression testing. Whilst I know the immediate answer is automation, I’m curious…
6 years ago I used to work in a group that took 2-3 weeks to regression test our releases, although working with a bespoke OS on a variety of HW platforms, much of it was waiting on imaging processes and disk array rebuilds. We had a group spend similar on all our software products too. Typically maybe one or two bugs catchable via regression test suites were released into the wild (most escalations were stuff around third party device integration or complicated networking).
We were told to automate these tests. Estimated several months worth of work and even then we’d still need some manual testing. I looked at the bugs found and they were all typos in readme docs, outdated links in our appliances or when someone went off script and found a bug that had been there for yonks. The number of “we broke something” bugs was tiny.
My proposed solution was to go risk based with just a week and smaller group, focused on appliances, installers & stuff we worried about… plus of course readmes. Over time we did add some automation to speed this up, ending as 2-3 days for 1 person (even automated… appliance updates were slow!) In the 5 years that followed the number of defects released into the wild went… down…
Moral of the story. Automation is the best way to run the regression tests but writing them isn’t free. Before getting all excited about automating your testing, make sure the tests are worthwhile first!
I use the RCRCRC mnemonic to help me identify the scope.
It’s there in the MoT Heuristics Cheat Sheet.
The Recent and Repaired are only things I’d consider around merges. My question here is that if a recent change or fixing a bug introduced the risk of regression, why wait? We should be targeting testing these risks immediately. I get that totally depends on team structures of course!
I guess that’s an interesting question. Are the teams that spends weeks on regression testing having to do so because they are too far removed from the development of a feature to have confidence in testing when a story was closed? Or it simply is t being done?
Amazing inquiry!
In our team, we always run regression tests for every release, but this is not the case for all the tests. We perform automated regression testing along with selective manual testing and pay attention to the recent changes and high-impact areas.
Most of the time, the tests are successful, but sometimes, they bring to light some “surprise” defects which have been missed by functional or integration testing. Those small catches are usually worth the trouble because they prevent production issues from happening.
We have definitely been through the scenario of asking ourselves “why are we still executing this test?” quite a lot 
. That is why we constantly audit our regression suite, removing old cases and adding new ones depending on the recent bugs or features.
So, on one hand, automation does help to reduce the time for regression, but on the other hand, test maintenance makes sure that the time consumed is still productive.
I can hold my hands up and say yes we were one of those taking weeks. Its a long story for us (probably an articles worth) but I’ll try and summarize the symptoms:
The biggest benefit was when that senior leadership team changed to engineering people.
Where we are today is we average about 1 week (some products less, and some products more) - with significantly less resource, and to be fair less products. Our automated test coverage is about 60-70% on average so we’re not bad but we don’t have the resource to continually maintain those frameworks - in dev or QA - so we prioritise those tasks accordingly. Our manual/exploratory testing is far more tactical now too. Still loads to do to meet the goals that @stuthomas set out in his test bash talk “Introducing Quality Engineering and making everything better”, but gradually there is belief that we can go quicker if we can collaborate from product to dev to QA better…but we still have business constraints that will hold us back.
So those organisation that are still taking weeks to regression test will probably have systemic constraints that others don’t. It could be the senior leadership team direction, the engineering budget, customer release constraints, lack of resources, reporting constraints, product complexity etc.
I likely used to torture junior testers by giving them scripted regression tests created by someone else to run. It may have been 20 years ago but I still carry a bit of shame.
Even when we automated them all there was still a level of mind numbing, tracking and maintenance.
Why did we do that. On one side it was a massive, complex and interconnected application, major releases were annual and often rewrote 30% of the product. This explains a small part of this though in no way makes it the correct approach.
The main reason though was we did not have a clue what was changing and the impact of those changes, there were no discussions with the developer before the code was written like what areas of the system this could impact, what are the chances of regression etc. Combining this with low developer unit and integration tests meant risk coverage was at a very inefficient layer. It was a cover everything approach and yes many people asked wtf.
Thankfully it’s over 15 years since I’ve worked on a project where regression has been deemed the most important risk, when good development practices are in place, regression risk is being actively discussed, developer tests are in place its just another risk amongst the many and you actually get more sensible time and effort allocations, say 10% for regression focus.
I still come across occasionally two common scenario’s when consulting.
One: those poor development practices have created a technical debt where regression risk takes developers away from building the product and it then takes priority over all risks.
Two testers themselves are not actually considering the risks, for example cross browser risks are generally fairly specific that could be covered with a small number of tests yet they run all there tests across every browser scenario, its likely more a case of just not thinking about it enough than a wtf though.
Its these simple nuggets that I live for! 
Totally agree that having those good practices plus dev & test closer (ideally intertwined) mitigates the need for so much regression testing!
@oxygenaddict Yes,
In my case, I worked on a fairly basic web application with both admin and user portals, but even then, regression testing took significant time. The application had multiple modules—login, user management, forms, reporting—and they were all interconnected across both portals. Any change in one area could potentially affect another, and sometimes changes in the admin portal would impact the user portal or vice versa. So we had to rerun tests for all major workflows on both sides each time, plus add extra focus on whatever was changed in that release.
Interesting. It sounds like an architecture design has had quite expensive implications! I take it that unit testing couldn’t help catch these sorts of regressions?