Here’s another tool that emphasizes change analysis (click ‘show original’ at the bottom):
1 Like
I really like your clear logical observation on non-regression bugs and sofas! People easily get confused by logical scope and pointing it out makes everything much clearer.
But I think defining regression bugs according to prior testing is the wrong way to go. The bugs exists independent of testing. It is true that if you do run the same test again, it should be because you are doing a regression test. (But you, or likely someone else, could have performed a test-task so unreliably, that you just think it has to be done again for the results to be reliable). And it is also true that it is much easier to classify something as a regression test if you have performed the test before and thus have established that the functionality worked at that time. But the bug exists independently of testing and whether it is a regression bug or a non-regression bug is also more or less independent of testing.
‘More or less’, because these classifications are made for a purpose. We classify bugs to help us test and develop better. And all regressions happen for an immediate reason: things do not magically change. It could be a change in code, infrastructure or usage. And if it is a change in code, the bug will always be in the interplay of the code that was there before and the code that is introduced. Determining whether it is a regression-bug or not is actually a placement of blame: is it the old code that should have been able to handle the new situation or is it the new code that should have used the old code properly or not have relied on it to handle the situation. An example with infrastructure: if the system does not work in the newest version of Chrome, is it because the code is not resilient to change (a non-regression bug in the code), or is it that Chrome has changed in unpredictable ways? If your back-end runs out of memory, is it a (non-regression) problem with scalability in the code or is it because the use of the system has changed in ways you could not imagine when writing it.
This ‘placement of blame’ is not about rubbing peoples noses in their errors, but helps the developers establish coding standards: When should they focus more on making sure their code is reliable for future features (and changes in infrastructure and usage) and when more on making sure that it is consistent with prior code.
This also defines a focusing on the test of the new code. By saying ‘Don’t worry about regressions; we have (automated or manual) regression tests taking care of that. Just make sure that the functionality you have implemented works properly.’ you allow the developers/testers of the new code to focus their effort in testing/development.
I have myself worked as a lone manual tester, where the distinction did not matter much: I continuously tested the whole system, with more focus on the areas my test-analysis told me were most likely to be touched by the new functionality. I did not have regression tests, because testing the same again and again would be daft, but I regularly got around in the whole system, either because my testing brought me there or because I had not been there for a while. But now I am doing automated regression testing in a place where developers test their own (and each others) code. In this environment, they need to trust me to ensure that the hard to predict errors due to strange interplay with (possibly buggy) prior code and changing infrastructure and usage are handled by my regression tests. This helps them focus their testing on the actual changes they have made and its scalability and resilience. And the fact that they know I am not going to do that for them, also helps them focus. We need to trust each other (and as a QA, I am a little more responsible for setting up a process where this works).