Analysis Tools to avoid the "Test Everything" Test Strategy

Hello,

My name is Damien and I am Lead QA working in Financial Services.

**I would like to know what analysis Tools or collaboration techniques teams are currently doing to help come up with a more targeted Test Strategy ? With the goal to reduce the number of tests written for a new feature. **

I am trying to change the mindset of our team/department, to move away from the “Test Everything” Test Strategy mindset. As a result our test suite is at 16000 automated Cucumber scenarios. Which is now difficult to maintain with a lot of duplication. When working on a new feature, we end up writing tests just to demonstrate that it works with an existing feature. I would love to find a tool or come up with a strategy, that can scientifically inform us that, we do not have to write these tests because etc.

I am keen to find out what teams are doing to help with the Analysis phase when working on a new feature.

Looking forward to hearing your awesome ideas.

Much appreciated.

7 Likes

Hello Damien,

I stumbled just last week across this paper about “Change Driven Testing” which might help you:

Also I am devoloping as a side project a tool myself which tackles exactly the problem you are describing.
The tool uses Git and static source code anylisis to determine which tests of the defined test suite(manual or automatic) should run in order to test the newly introduced changes. I am still not done with the development, but I can provide a prototype if you are intrested in trying it out. The tool works however so far just with Java applications.

5 Likes

That tool sounds interesting Samuel, I don’t have a java app but I would be interested to follow your project. What is the best way to follow updates on it?

3 Likes

Hmm to be honest there is so far no way you could follow my project😅
However anyway I thought about putting a beta version online to get feedback from the community and inform about updates for people like you who are interested :slightly_smiling_face::slightly_smiling_face:
Sounds like a little project for the weekend😉 will keep you informed

3 Likes

Hi Samuel - “Change-Driven Testing” sounds really interesting!

I have read about Test Impact Analysis previously.

The ability to only run those tests which test the code which has changed sounds like a great addition to any teams CI pipeline.

Test Impact Analysis complemented by Test Gap Analysis sounds extremely efficient.

I will have to read up on this more!

Thank you for sharing this.

Damien

3 Likes

@damien.gault these “16000 automated Cucumber scenarios.” are small test cases where you are validating 1 step as 1 Test case.

I will suggest to have End to End test case that will reduce your count and you can easily analyze report and conclude the release health.

2 Likes

@samuelm do you know of any tools which can help with…

  1. Creating Treemap visuals of your code
  2. Test Impact Analysis
  3. Test Gap Analysis

I’d love to know if there are existing off the shelf tools that work with Java.

Thanks

2 Likes

Hi Damien,

I think Teamscale does it. I did not try it out myself and I don’t know if they support Java, but I think they do.

Here is the link:

If I got it right teamscale is developed from the same company which put out that paper which I sent you some days ago.

Hope this helps​:wink::wink:

1 Like