Defining 'regression' in testing

A sofa is a non-regression bug as in “¬(regression bug)” but a sofa is not a non-regression bug as in “¬(regression) bug”, because a sofa is not a bug. Probably not a bug.

Regression requires testing two times. Once to notice a problem or notice the lack of a problem, then once again to notice it (again, or for the first time). That means that any software I’ve never tested before doesn’t contain regression bugs. Unless we’re saying that anyone could have tested it before, in which case a regression bug is one where someone decided there wasn’t a problem with something according to some model and then someone else or the same person decided that

That means that a non-regression bug must be one in something nobody has ever tested before (how something gets built without anyone doing any testing of any kind is a pertinent question), or, for bugs that have been long in production where it’s never worked, one where we must decide that the testing we did do didn’t find the problem and our coverage against whatever model we had would have been sufficient to find it.

So… a bug of regression is where via a check or test a human discovered there wasn’t a problem, then another human (or the same one) discovered there was, where the human in the latter case decides that the original check or test would have found the problem they just found. OR the timing order is reversed and we look for problems that would have existed if we had looked and found that they didn’t (according to some model), in an earlier version.

A bug of non-regression is where the latter human decides that any check or test done by any human that they know of either detected the “same problem” (via whatever criteria we’ve chosen to describe it being the same) or did not detect the problem. Or everyone only tested once, and knows what each other tested and the coverage they achieved.

Interestingly that means you can turn potential regressions into potential non-regressions by increasing the epistemic risk gap.

Also interestingly if the bug is later in time within the same version would we call it a regression? Is something a regression if it goes wrong because the environment changes? Is there really such a thing as a regression bug, or just a regression “vulnerability” (in the sense of something that permits a problem under the right circumstances). Should categorise by causes rather than symptoms?

I don’t know what point I’m making, but I’m enjoying myself.

2 Likes