Puzzled about Retesting and regression testing articles

Discussions
, ,
1

I’ve had an interview question on TestGorilla where the following were expressed:

  • Regression testing ensures the recent changes in the software code have not affected the unaltered sections.
  • Retesting establishes that the failed test cases are passed after the defect is fixed.
  • Regression testing includes no defect verification, retesting does.
    True / False

Now searching about ‘retesting and regression testing’ on Google I get dozens of sites that mention similar statements.

But, in my head, I have so many issues with this, and it’s so strange because I find almost nothing stating otherwise or contradicting these.
Especially since these days a lot of interviews rely on popular questions that you find on a Google search, I’m worried I’m being left behind somewhat if I don’t study that alternative path.

Here are my comments:

  • ‘regression testing ensures’: it does not ensure anything, testing doesn’t show the absence of bugs, but searches for their presence.
  • ‘regression testing … recent changes in the software code’: changes can happen without touching the code, the system, or the software - isn’t testing that happens for those changes also regression testing? and what does recent even mean, how is a ‘recent change’ different from a ‘change’ - you don’t do regression testing unless it’s recent?
  • ‘have not affected the unaltered sections’ - is regression testing not covering changed areas?
  • ‘retesting establishes that the failed test cases’ - getting over the ‘test cases’…is retesting limited to the same ideas as before being blocked?
  • ‘retesting … after the defect is fixed’ - can’t you retest without a defect fix? what if you want to study the area more deeply, with new tools, or vary the data, to uncover new defects?
  • ‘retesting establishes - test case passed - after defect fix’ - it’s not ‘established’, as we wouldn’t know if a developer’s defect fix actually fixed the problem or didn’t introduce a new one in the same changed area; and the scope of retesting seems to be the setting of a test case to pass, and I wonder why it’s not to actually search for trouble; what’s the technique used to test for ‘defect fix’ if the scope for retesting is to establish the test case to ‘passed’ after the defect fix?
  • ‘regression testing includes no defect verification, retesting does’ - so if I do regression testing and find bugs, my regression testing stops, I do retesting then, then again regression? isn’t regression testing repeating testing(retesting?); retesting can include no defect verification as well in case I do it for different reasons;

Bach and Kaner in a 1999 paper(Paradigms of Black Box Software Testing) mention this:

  • Regression testing: Repeat testing after changes
  • Goal: Manage the risks that (a) a bug fix didn’t fix the bug or (b) the fix (or other change) had a side effect.
  • Paradigmatic cases: Bug regression, old fix regression, general functional regression

Another large paper on Regression testing from Kaner:

I like Kaner’s and Bach’s views very much and it’s what I’m thinking of when repeat testing(retesting) and regression testing pop up.

3 Likes
2

In my experience, I have understood Regression to be an exercise that is executed (usually by automated means) to provide confidence that any changes made hasn’t impacted the system in a negative way.

Re-testing (or Confirmation testing) is when re-testing a resolved defect from the development team by following the exact steps to recreate that was stipulated in the original issue. Ideally, the issue then can be ‘closed’ or ‘failed retest’ if the issue still persists.

3 Likes
3

This is something that happens a lot in testing, there just are no terms that are universally understood to mean the exact same thing across people and companies. And for some of the terms like regression testing and retesting they are also used to mean the same or similar thing.

My way of managing this is to be sure that we in the group I work use the same term for the same thing as much as possible. And that I know what activities a term might refer to. I also try to understand the purpose more than the specific term. In the case of bug fixes there are two things that I would be interested in, testing that the fix fixed what it was intended to fix and testing that the fix did not introduce new bugs. It just so happen to be the same two things I look for every time an existing software gets updated, be that from adding or changing functionality or fixing bugs. What tend to change is the scope of the two parts. A common term for this is regression testing. And if you are a team that will work together on the software it might be good to distinguish between the two parts so we can work in parallel and still be sure that we cover different aspects of the testing scope.

But I have seen regression testing be used as: Running the automated tests. The scope of tests that we always run no matter the scope of the change. The tests we run as a response to a change.

Retesting as a term I have most commonly seen used as “re-running” failed test cases from before. It is also commonly used in organisations that have an unhealthy relationship between testing and developing where communication is done through tools and documents instead of between intelligent persons so I tend to be extra sceptical when I hear the term. This would be the opposite of context based testing, i.e. we do things no matter what have happened and the situation. And it tends to be error prone and slow.

6 Likes
4

Like @ola.sundin explained. Regression testing and so many terms are easily miss-used. Poor human-human communication is a know cause of defects, so assuming that you know what a person wants based on the word they used is dangerous for us testers. As your company grows by acquisitions or by new hires, the actual meanings of a lot of the terminology needs to be socialized (yes another poor word choice), or re-explained to everyone in the company from time to time. If not, you may find entire teams happily use some terms differently based on their domain. And that’s OK for different teams, as long as everyone is aware of the dialects that different teams use. Dialects sometimes just have to exist, for example when working with support versus developers and at those points, choice of words matter.

3 Likes
5

@nwheelhouse has succinctly described the difference.

It doesn’t surprise me that in a skills assessment setting, the wording is massively over-complicated and not in plain english - those sorts of things in my opinion always ‘over-egg’ something that is really simple.

@conrad.connected makes good points about the use of language and terminology within teams though - again, if your team has had a ‘ways of working’ type of meeting at the start of the project, the naming of test phases can be cleared up here.

3 Likes
6

Your thirst is mine, my water is yours.

I suffer the same fate every time I turn to Google for answers on testing. Metrics, test cases, glossary definitions, whatever, it always turns up the same junk written with an unearned sense of authority. Freedom comes with a cost, certainly, but also with rewards, and I try to practice gratefulness for that.

I think that you’re right and they are wrong. I personally don’t think it’s a case of differing perspectives or terminology - I think that they have simply stated something incorrect with confidence, and you have made excellent and compelling arguments to the contrary. Don’t lose hope; I write the same replies once a week all over the internet to the same thinking, or form a good argument that is flat-out ignored in replies, and it sometimes brings me to despair, but all you can do is look to your morality and try to remain steadfast and attempt to be intellectually honest and, much harder, still open to learning something. Resist becoming jaded.

Testing is a weird social construct that often fights fear with ignorance, ceremony, traditions and talismans. Getting people to let go of their wards and protective charms is difficult - you’re asking them to become upset. You’re trying to sell the idea that reality is scary and we have to face it head on. It’s not surprising to me that there’s an industry geared to the sale of wards and charms. “Ensures”, “establishes”… well I feel much better. Automation can solve my regression problems? It can actually test for me? I don’t need to worry about the humiliation of getting something wrong I already got right? O rapture, drunken deep of joy have I, and I will not taste the bitter of truth! Hold thy sharp, surgical tongues, wise men, from the crystal delicacy of my effervescence, as I float aloft the realms of lesser mortals! The mechanisation and over-simplification (bullshit) of scientific qualitative research (testing) is a big obstacle to the progression of thinking and discovery in our industry.

I find almost nothing stating otherwise or contradicting these.

There are few gems in the fields of rocks

I’m worried I’m being left behind somewhat if I don’t study that alternative path.

That is a good feeling. Not in the sense of being comfortable, but in the sense of utility and honesty. I think it’s okay to approach something with an open mind, but reject it if it fails to meet our standards. That is what medical science has done with homeopathy, for example - it has been repeatedly tested and found to be ineffective. Studying homeopathy has interesting utility, showing a time in the history of medicine where receiving no treatment (homeopathy) was better than receiving treatment (mercury poisoning), and we can study that without actually believing in homeopathy.

With regards to interviews, I cannot go into an interview and lie about what I do. I’m happy to discuss questions I disagree with. I’ve pushed back on questions and not gotten the job as a result - but I think both of us benefitted from that because I want to be valued for my skill and knowledge and they are going to have to put up with my lunchtime workshops and process improvement projects.

I think an interesting route to explore would now be: How do we do regression? What heuristics can we employ to build a regression strategy that’s sensible? Then we can discuss risk mitigation, change risk, opportunity value. We can finally say “trying to repeat ourselves reduces the variability that finds problems”, and better allocate our resources.

Live and drink, friend.

3 Likes
7

Kaner and Bach indeed didn’t make that distinction, but I’ve found it useful in conversations to have a term (retesting or confirmation testing) to refer specifically to confirming a bug fix had the intended effect separately from a term (regression testing) for testing whether a code change has had unintended effects where changes were not made.

Especially since these days a lot of interviews rely on popular questions that you find on a Google search, I’m worried I’m being left behind somewhat if I don’t study that alternative path.

In interviews I don’t ask anyone to define software testing terms, but if it were to come up, I would actually prefer a person to show (like you’ve done) that they’ve thought about it and have passion about the craft rather than a person who provides perfect textbook definitions.

As for some of the problems you see with the terminology, aside from the ones where it’s just online sources using imprecise language:

‘regression testing includes no defect verification, retesting does’ - so if I do regression testing and find bugs, my regression testing stops, I do retesting then, then again regression?

“No defect verification” is not equal to “no defect identification”.

the scope of retesting seems to be the setting of a test case to pass, and I wonder why it’s not to actually search for trouble

You’d do that as well. I would consider that part of retesting. Maybe some wouldn’t, to me it doesn’t matter as long as we use consistent language within teams and risks that should be mitigated are mitigated.

1 Like
8

Thanks for the healthy dose of sense, everyone.

The TestGorilla assessment that I received for an interview contained many similar questions with unknown terminology where you get to answer with True or False or pick a single correct variant.

It reminded me of ISTQB tests, except that in TestGorilla you have 12 questions in 10 minutes. ISTQB gives you 70 minutes for 30 I believe(non english speaker). I keep the ISTQB out of my CV as it’s an embarrassment to the craft.

I was not trying to stay specifically on just these two terms ‘retesting’, ‘regression testing’, which were just an example, but about the whole idea that there are different worlds of view that I’m not accustomed to.

And I wondered if my career development in testing should also go in that direction, as most of the world/companies go in that direction. I might be jobless otherwise in a place where you see 3 new job openings per month only.

And I thank Chris for the sanity review and kind words :slight_smile:

2 Likes
9

Interesting read, I had never really thought of them as two different terms but when I think of our environment they definitely are. I also find when you say “Regression Testing” to anyone (especially a Manager) they have a look of fear and ask “How long will that take?”. Whereas if I say “Re-testing” they don’t look so concerned and ask “Are you just being thorough?”.
In our Team, we test bugs and/or features (in test branch builds), regression test the areas impacted by the changes and once we have a master build we re-test the changes (at a high level) to make sure nothing has failed in the merge/build process. This process does require some good test scenarios and knowledge of the risky areas.

10

In my team, regression testing is a verification that everything’s still working whereas re-testing checks that the fixes have worked and haven’t broken something else.

A full regression cycle can potentially take weeks in a non-automated project; I have one project that requires about 10 person weeks for a basic regression test - 25 plus years of code and hundreds of configuration variations so no chance of ever completing a full regression test (partly because much of the system doesn’t have specifications or test cases and it’s unlikely anyone remembers enough about the original undocumented requirements to be able to cover it all). The aim on this project is for regression is to make sure the most common user journeys work as expected and there are no nasty surprises lurking.

For every change we make we test, check the fixes and then re-test the new features as a final walk through to minimise risk.

1 Like
11

I have one project that requires about 10 person weeks for a basic regression test

If I hadn’t already finished my tea, it would be all over my screen right now.