Handling test failures in test automation: Optimizing the tests vs sporadic performance of the web app?

In my project we have a playwright test suite that runs tests on 3 browsers (firefox, chrome and webkit)

Not very often but at least once every 3-4 weeks we see the tests failing on one of the browsers for an unreproducible reason (sometimes playwright is a bit too quick for the web app and thus fails). Very often, re-running the suite for the specific browser would pass the tests.

Now this is something I’m pondering over, I can try to spend some time optimizing the code to get the tests to run green unless the web app develops an issue (which would be a concern for us).

But my concern is that I know for a fact that the software still has room for improvement when it comes to performance as whole and I also know that the goal for automated tests is to fail rather pass so that we get the chance to improve our software. Surely a short string of failures shouldn’t make the tests unreliable?

1 Like

I’d tackle it through approaches/angles:

  • Do I have enough time to spend analyzing or experimenting an interesting failure, then why not dig deeper to see what/why’s failing;
  • Is the place/failure one that would matter? if not maybe just delete or rewrite the flow;
  • Is the failure consistent? Can you find the pattern? I’ve seen consistent ‘random’ failures at specific times. Turns out it was a server rebooting process.
  • Is the failure interesting? There are many of those and no time to check? flag them somehow to mark them as unstable(they can reveal interesting timing or browser-specific bugs); check the behaviors failing yourself now and then;
  • Build a general process for this kind of problem; Many companies once they scale end up with hundreds of these, and they throw in more automation engineers at it. If the automation is of low importance, outcome, treat it as such don’t waste too much time and be efficient in finding another solution to a problem you’re trying to solve
1 Like