How do you approach regression testing in Agile delivery world?
We are delivering software every 2 weeks and when Sprint testers do targeted regression for 2 days. Its from big part manual, form other part automated but still we fill those 2 days by testing. What would you do to test release of huge product in 2 days as effectively as possible?
Short answer is: test the most important parts, as much as you can do in 2 days and communicate well to the management the situation that not everything has been tested and they need to do something about your lack of time.
Longer answer required more details, why did you choose to ask this question in the first place? What is the problem you are trying to solve (I know you wanna test your system, but why did you ask this question now)?
For long answer. I want to shape our regression in the way that regression brings value to us. The aim of regression to me is not negatively influence crazy complex features already running in production. We work in Scrum way with teams delivering features by the end of the sprint tested and ready to ship. Automated regression tests are also part of feature delivery. Ideally, I’d like to see regression tests passing in Jenkins and release it. But due to the complexity we work with, we are unable to automate all the regression testing, its quite technical challenge for us. So we have extensive manual regression suite alongside with our automated regression suite.
So I’m searching for ways how to do regression in such environment effectively, because we won’t to test full regression every 2 weeks.
I’m working with customer support and product team on incorporating short answer.
The aim of regression to me is not negatively influence crazy complex features already running in production.
That makes sense.
How many bugs did you have in production per release in the last 12 months?
Once we know that, we should categorise them by funcionality group and impact. Then, we can proceed to making a better informed decision on where to conentrate your efforts.
If you do not have many bugs in production, we can approach it differently, by risk areas with high impact for example.
I like this reply a lot.
If you have the luxury of knowing what fails in production, are able to monitor that environment closely, can do root cause analysis of each problem and are able to draw meaningful conclusions from this information, I’d take a strategy based on that knowledge above almost any other strategy every time.
(Risk analysis and testing for unknown unknowns can complement that strategy heavily though.)