You Broke My Tests!

🙋 Questions
,
1

How can I reduce this problem?

Context:
10 scrum teams with devs and testers in each, in same ART, doing SAFe. All contributing to a monolith ecom site. Weekly global prod deploy. Testers from those 10 dev teams maintain a suite of ~ 2500 Selenium checks that run daily and mostly work (e.g., they find bugs and have a false positive rate of about 5%).

Problem: The false positive rate fluctuates +/- 5% because changes to the product-under-test by Team_A often break Selenium checks owned by Team_B and Team_C. Fixing the breaking tests is unplanned work.

2 Likes
2

Hm interested dilemma! Here some ideas:

  • Definition of Done: make it obligatory cross-review by teams
  • Test Review Board thru pull requests: make a new process where everything goes thru pull requests. Thus TeamA reviews and merges TeamB, TeamC and vice versa
  • Impact Analysis: make correlation between tests, check impact once changes are done, prior alarm system before execution
  • AI: makke self-healing tests Self-Healing Tests | CodeceptJS
  • Motivation: make bonus for each team based on false positive and they will come up with some ideas you don’t expect :wink:
1 Like
3

Adding a few cents to Mykhailo’s statements:

  • The rotated release team which is responsible for ensuring that builds are green each week.
  • Automated impact analysis to run only impacted tests and not all 2500.
  • Shared responsibility for the tests.
2 Likes
4

Step 1: Move it from Unplanned work to Planned work in your next sprint to bring it to everyone’s attention.

Step 2: Try to separate CI suites that run on daily, weekly and release level. ~ 2500 Selenium checks daily feels anti-pattern to think of. Maybe, you have a reason for it. I am not sure. Some optimization or moving test to API layer needs to be done.

Step 3: Observe if it improves the situation. Time frame: 2 sprints atleast.

2 Likes
5

Are the tests breaking or failing? I think this is an important distinction to make, as if the tests are actually failing but being treated as broken, then you may end up “fixing” a test so that it doesn’t actually do what you think it does.

Either way, I agree that this should definitely be part of the planned work, and think that an investigation into why this is happening could be helpful.

Are the tests flaky?
Is the code difficult to maintain?
Is your branching strategy contributing to the problem?
… And so on.

This is definitely an interesting scenario. Let us know what you end up doing!

1 Like
6

The problem is that end-to-end tests are remote from the locus of the problem.
In a team with mature development practices the E2E suite only serves to ensure the ‘walking skeleton’ is maintained with integration, component, contact and unit tests doing the heavy lifting by inducing through their test signals a design that has high cohesion with low coupling.
So how to fix ? Committing the developers to maintaining a suite of contract tests may help, but that can de difficult if there’s no skills adoption culture. I think as @parwalrahul says moving it from unplanned to planned work would be a first step - that surfaces the problems for everyone’s attention.

3 Likes