Hey all, this is a question I know that has been asked before, and I know is a contentious issue that some disagree with. But…
What metrics do you think are useful to measure the features of your testing?
I’m not looking from the view of senior stakeholders, but from the view of perhaps a principal engineer/test lead (or similar role who is accountable for the testing strategy of a company/division/function). This is something I’ve been thinking about for a while. I think what helps perhaps, is to tie it into a mission statement from testing. So for an example, my mission statement would be something like:
Continually deliver working software that meets customers needs, by focusing on preventing defects and providing fast feedback.
On the back of this, a couple of metrics that come to mind for me, are:
- 10 day rolling bug/live issue count (plus which way this is trending) - This for me is the ultimate metric that defines whether testing is meeting its purpose or not.
- System uptime
- Build times (measuring how quickly we can give that feedback as per the mission statement)
- Story run time (How long does a story take to be developed from inception to deployment - the goal of testing is to catch bugs, but a feature of that is how quickly we can do it. If it takes 2 years, theres perhaps a problem here, if you could see this across teams, it might help to see a bottleneck)
- % of broken builds - (hoping to visualise here how efficient our build process is, are there flaky tests, are there common failures and if so could they be prevented)
- Mean time to resolution - (perhaps a measure of the average time a build is broken for, trying to improve the efficiency of diagnosis of issues, if it takes 40 mins to decipher a test report and figure out why they failed; that’s perhaps an issue)
I’d be really keen to know what other people are capturing and how that helps them aid their testing. I don’t want to just capture metrics for the sake of it, or to report to management. I want to drive improvements going forward. I think that’s perhaps why its important to distinguish between the goal of testing (defect prevention) and the features of it (how quickly we can do it, how much feedback we can provide to developers)
Love to hear your thoughts and opinions.