Reporting on automated test failures, reasons and trends - best tools for aggregation

I wondered what people were using (if anything) to track trends in automated test failures. I’ve taken a look at Extent Reports, Report Portal and Cypress Dashboard as potential commercial options.

We use a mix of Javascript / Java based tooling (Selenium/Cucumber/Cypress/Playwright), Jenkins as a CI server and Jira as a defect tracking tool.

As a minimum I’d like something that can represent:

  • pass rate/fail rate for test/scenario/pipeline
  • show this per e.g. week, month or however fine grained

The end goal is to get a clearer picture of test flakiness so to help inform which tests need attention / quarantine etc…

I’m inclined to just used Excel for now to try focus in on what metrics we want to cover and experiment with some charts. Ultimately, it seems there should be better solutions out there that integrate with existing tools and minimise the amount of manual effort.

Just wondered what people are finding effective and if there are solutions out there I have not thought of.


This is a topic that I am also very interested in.

For all of the discussion out there around test flakiness and automation efficiency, it sure seem like the generally accessible tooling available does a poor job of tracking these things over time.