I’ve been thinking about quality metrics and how they feed into health metrics in other parts of the business. I recently wrote a post about how metrics can be misleading (link in profile if you want to check it out), and I’m curious whether this dynamic is just everywhere?
I’m worried it might be (hoping I’m wrong, though!). Has anyone figured out how to avoid increased dashboard visibility creating pressure to sanitize metrics or make targets more “achievable”?
I’d love to hear about any practices that have worked for you! (Stories about spectacular failures also welcome )
In a nutshell, if your metrics are not helping to make better decisions, they’re not the metrics you should be tracking.
I’m not sure I understand how increased visibility would create pressure to sanitise - if your dashboards aren’t seen/used/discussed, then what are they for?
I’ve written a whole (free) eBook where I discuss how I think metrics can be used for improvements in QA.
I’ll also teach a workshop at the TestBash in Brighton, which is focussed on. QA metrics for smaller teams.
I’m a stats nut. Love them, throughout my leadership career I’d been brought into the value of KPI’s…but not necessarily good ones. So I’d dive into data and bring out what I thought were KPI’s, loads of them. I’d got in front of people and shared them, talked through, sanitised them and got a dashboard up and shared.
But…
Over time it was clear there wasn’t the buy in or improvements. So I took a step back looked at them and asked “If the trend goes up or down, is that a clear indication of improvement?” - and in most cases it wasn’t. They were interesting bits of data that whether the trend went up or down, it didn’t actually affect the outcome.
So I trimmed them down, had the same conversations and relaunched them to far greater impact. They appeared to matter this time and get a reaction - sometimes positive, on some occasions not so positive. But one other thing I think its OK for them to change, as the business changes, what is important may need to adapt or reprioritise.
I’ve come across this a lot, actually. Maybe “sanitize” is the wrong word, but I’ve often found that the same metrics that teams use to make better decisions internally tend to end up being used in other places with different intentions (executive reporting, or worse even team comparisons). Bc of this, teams become reluctant to set targets that actually represent value to users and instead choose targets that are achievable.
This creates a disconnect where the dashboard shows green, but real conversations about quality happen offline. In my blog post, I wrote about an example I came across at Dropbox with Paper - product dashboards looked impressive (and if any quality metrics made it there they were the ones that folks felt good about sharing upwards/outwards). But the quality metrics our team measured and championed were not, and attempts to improve the visibility of those were pushed back against because that would have made the work/decisions folks needed to make harder.
Sounds like there’s a whole bunch of different problems to fix before you want to fix the metrics.
If the company culture is broken, no amount of great metrics will help your team deliver in a healthy way.
Agree with this statement that metrics should not mislead us instead, they should guide us in improving the quality.
Also, there are different types of metrics being observed, which are being calculated through different ways, like dashboard, program output, however
For automation, one of the metrics we use is :
Test coverage% % =
(test cases automated )
----------------------------------------------- X 100
(total test cases - cannot be automated )-- This is the total number of automation test cases
Yep, agree! That’s why my question was framed more around the culture around metrics (and not metrics themselves).
Maybe a better question is: in environments where there are similar cultural challenges, how do you maneuver those while also maintaining meaningful quality standards + radiating that information throughout the org (as necessary)
And is my experience really an outlier? I’d love to hear that if true - especially as I’m searching for my next role
Your story really resonates - that “clear indication of improvement” test is the kind of discipline that I think is important to have in metrics processes. At my last company we actually had a pretty solid metrics process (sans those cultural challenges).
What I found fascinating (and frustrating) was how the same metrics that drove really great internal decisions would get watered down when they moved up the org chart, and teams would start choosing targets that looked impressive externally rather than ones that actually represented customer value.
And +100 to your point about metrics needing to evolve, stale metrics don’t help anyone either and just add to the noise.
)
Added to my reading list!