Hi all. I’m currently working on a Customer Reported Defects Metric report where the management team would like to see new defects reported by customers each month, as well as those that we close and retain as backlogs on a monthly basis. Now one of the steps that we do internally is to try to find out if any of these issues are existing in versions supported other than the version/release where it is reported, and then make a clone or copy of the bug for fix in those other versions. Should I include those “copies” in my count for the defects metric/report?
For starters, to be a clean metric, these need to be “Uniques”, frequency of identical report only tells you that a component is “customer-facing”, which is another reason this metric is subjective. Fixing high frequency bugs with low impact becomes a game.
Identical reported bugs in different versions of product are a software development lifecycle (SDLC) problem. It is useful to know if a bug is a regression and not “introduced”, but in my experience it’s only worth the deep analysis and time needed to “test each version” when the bug is hard to reproduce, or the dev team is overloaded and cannot identify the bug source quickly.
Moreover. Once you only count Unique Customer Reported Defects, you get C.R.U.D. Basically any metric will measure only the thing it wants to, and no amount of measuring it again will actually resolve defects more quickly. Better to spend time ranking bugs using an agreed upon severity, and then using a TTF or time-to-fix metric for all P1 (priority 1) S1 (severity 1) bugs.
Just a little side note, the popular bug tracking tool Jira, intentionally does not have a bug severity field by default, think on that. Very good question though @hmwcoffee . I hope my hating of metrics does not put you off though, I’m not wanting to start a conflict, just to give you an exit route.
Testing multiple versions for the same bug is not just an SDLC problem. When you have a need to support multiple clients who are using different ‘trains’, multiple pipelines need to be tested even in Agile.
It helps to highlight if multiple versions (and which versions, in the bug report), so that heartburns with other customers can be avoided. But I agree that count by itself does not make sense.