I wondered what people were using (if anything) to track trends in automated test failures. I’ve taken a look at Extent Reports, Report Portal and Cypress Dashboard as potential commercial options.
We use a mix of Javascript / Java based tooling (Selenium/Cucumber/Cypress/Playwright), Jenkins as a CI server and Jira as a defect tracking tool.
As a minimum I’d like something that can represent:
- pass rate/fail rate for test/scenario/pipeline
- show this per e.g. week, month or however fine grained
The end goal is to get a clearer picture of test flakiness so to help inform which tests need attention / quarantine etc…
I’m inclined to just used Excel for now to try focus in on what metrics we want to cover and experiment with some charts. Ultimately, it seems there should be better solutions out there that integrate with existing tools and minimise the amount of manual effort.
Just wondered what people are finding effective and if there are solutions out there I have not thought of.