An important aspect of a test report is metrics selection and inclusion. These metrics provide valuable insights that allow stakeholders to gauge the effectiveness and efficiency of testing efforts. In your experience, what essential metrics should be included in every test report?
4 Likes
An important aspect of a test report is metrics selection and inclusion.
Is it, though?
Reporting is about getting the right information to the right people when, where and how works best to meet our goals. I don’t think that metrics have a forefront, default role to play in that. I think that most people don’t understand metrics, metrology, statistics, statistical significance or psychology enough to be brazen with collecting and reporting numbers.
Many - perhaps most - of the great issues of science are qualitative, not quantitative, even in physics and chemistry. Equations and measurements are useful when and only when they are related to proof; but proof or disproof comes first and is in fact strongest when it is absolutely convincing without any quantitative measurement.
Or to say it another way, you can catch phenomena in a logical box or in a mathematical box. The logical box is coarse but strong. The mathematical box is fine-grained but flimsy. The mathematical box is a beautiful way of wrapping up a problem, but it will not hold the phenomena unless they have been caught in a logical box to begin with.
— John R. Platt (1964) “Science, Strong Inference – Proper Scientific Method (The New Baconians)”. In: Science Magazine 16 October 1964, Volume 146, Number 3642
One could look at inexpert measurement as a way of placating the fears of the uninformed as the exact opposite of what testing is supposed to be for - breaking illusions about the software.
In your experience, what essential metrics should be included in every test report?
None. Every test report is different. Some are one line, some are books. Some are casual mentions, some are white papers. Some are spoken, some written. Some are about the safety-critical, some barely matter at all. Some are secure communications, others are public knowledge. The test techniques vary, the findings are different, the author changes and the readers can be diverse. There’s never one specific thing to include on a report.
Many metrics can be used to sell any political stance you wish to take. The number of bugs found has increased? That’s terrible, developers need to be more careful. That’s great, testing has become better at finding problems. That’s expected, this part of development is more error-prone. That’s because a new client is making a lot more fuss than previous clients. Our values as a company have changed, and because of a shift in what’s important we are reporting new issues in line with those new values. There’s so many real-world implications we can clumsily draw from the simplest metrics.
So, if we’re going to use metrics, we use them carefully, with humility and understanding, and do not risk them being misunderstood, gamified, or used beyond their limitations by anyone. We take responsibility for our reporting, both morally and in terms of accuracy, as best we can. We also use them only as and when they will make things better. Metrics should be treated critically, and they should drive questions, not answers. The same as any fallible oracle we employ in any other area of testing.
Then we are equipped to make case-by-case decisions about metrics and their reporting.
Edit:
I’d like to add something about measurement, and an obsession with turning it into quantitative mathematics. We measure as testers, and as human beings, all the time. We make evaluations about things that are happening and use that to help decide on a course of action. Your soup is too hot, you let it cool so it doesn’t burn your mouth. This does not require any numeric measurement of the volume of the soup or its accurate temperature to +/- 1 degree Celsius. And so it is in testing - we can drive real change with real observations, whose validity can be borne out in repeated and carefully controlled experimentation to an exacting degree, without a single number written down. It’s okay to make the numbers work for us in a sensible way, as and when we want, without feeding a constant addiction for certainty with the methadone of metrics.
6 Likes
Hard question which I am going to respond with a simple answer: It depends.
Had projects where the only metric the stakeholders cared about was amount of test run and how many passed or failed. We basically had to have all tests passed before doing a release. And that can take a while if you have 15000 tests ( even though they were all automated).
I had other projects where tracebility was key. The stakeholders wanted to see what requirements were covered and what is the status ( bugs, new features, still in progress works).
In other projects only the amount of bugs were important. If we had more than x bugs then no release. We did try and do quantifications in the sense that we would release if there are no blockers, less than x criticals and less than y major or minor. But that lead to discussions with the stakeholders because for them all tickets are critical or blockers 
If I had to do it a way that I wanted I would do a mix with tracebility being in the center combined with some UAT testing results.
In a team building day few years ago I learned a hard lesson. Managers mostly get swamped by miss-information from their minions and from other sources. They don’t like laborious detail, and are actually used to being fed garbage by minions. Managers thus often have to act on very little real information. Managers only love metrics, because their CEO loves metrics. I was shocked, horrified, and although I felt sorry for the managers, all I could do was never ever ever again write a long wordy report again. Let them ask the questions, listen carefully to the questions, if they ask none, change nothing. If they do ask, come with solutions to try, not with yet more problems.
Last week we had a release readiness review, so I had to put some metrics down. We had a very low test fail rate the night before so I showed off the test reporting it as 90% passing. But was clear that its normally around only 80% right now due to a lot of intermittent issues because we added a new test node and a few new tests for a feature that is still in-flight. The big risk was that in-flight work was impacting our branching strategy and thus our bandwidth. The new test node is my strategy for reducing that risk. Thankfully our process is not metrics-centered but around release frequency. If there was one metric I would value more highly than the automation pass-fail rate, it would be the “man-hours” spent in total in interactive testing per release cycle, with a view to bringing that down over time.
2 Likes
An excellently worded reply. I agree with you entirely.
There are lots of different types if metrics. But often “Metrics” refer to traditional methods of testing…test cases, test coverage, pass and fail rates etc. I even find test reports themselves a little “old hat”. I only write those when i think someone might actually read it, or there are the highest levels of governance required.
I haven’t provided a single metric to the upper echelons of the business in nearly 4 years and the world hasnt stopped turning. Simply, no one has asked. Features have been delivered and if bugs arise after release, we backlog them and prioritise accordingly. In a devops world, far less emphasis is placed on all that stuff. It has its place but it is far less important than once it was.
HOWEVER…
If you are a performance tester… then things are a little different. Metrics are your friend. If you don’t measure it, you can’t improve it. Etc etc…
That’s a whole different topic though. 
2 Likes