The topic of metrics came up in TWiT today and it seems like a good opportunity to define what metrics mean to us individually. We often think about them in different ways and how we think about them can depend on the context too.
If you had to define the word metrics, in relation to software testing and quality engineering, how would you describe it?
Metrics is basically a parameter based on which we decide the quality of software. There are many use cases of Metrics but most importantly this is being showcased to stackholders and leadership to display the progress in testing.
However, as we have different approaches in different types of testing hence their metrics also differ accordingly.
For e.g. Manual testing has different metrics and the team involved in that testing uses those metrics to define the quality, like number of test cases passed or failed or not executed blocked., number of open bugs, pass/fail ratio, etc.
When it comes to automation metrics related to man-hours are most important as we have to justify how many man-hours we saved with the help of automation which will eventually decide the ROI from automation.
Similarly, performance and security testing have their own metrics.
A metric is an objective measure of an area of interest that can affect software quality outcomes.
I have a mountain of āit dependsā caveats around that statement as its no silver bullet just to have metrics. They help, the āareas of interestā will be different project to projectā¦and software quality isnāt 100% āobjectiveā. If it was, weād have that silver bullet.
First we need to make the distinction between āmeasureā and āmetricā. In testing we measure many parameters like # of bugs; # of Test Cases; test effort in MD.
Metrics are measures that are tracking a process, assess performance, check for progress. Itās measurement with meaning.
If I measure # of bugs found; the metrics could be bugs per test effort; ratio of Critical/major bugs; # of fail test cases. All these could be measured by day, by week, by sprint, by project. They can be used to compare between sprints, between projects.
You can also have metric on % of test automation with an objective to increase by xx% in the next months, quarterā¦
Such metrics that have goals are often described as KPIs. Some donāt necessarily have a goal but can be tracked by Teams to see if the quality of their deliverable is improving or not, as an example mean time to resolve/fix bugs.
At my company we have many quality measurements and some good metrics but as mentioned by ghawkes, no silver bullet! We have a dozen of QA metrics, but nothing that magically can tell us if we can ship in production or not.
Are you sure itās a parameter and not multiple parameters?
Are you sure quality is singular and you donāt have multiple dimensions of it?
I completely agree with your point. Metrics are not just based on a single parameter but a combination of multiple factors. Also, software quality is not just one thing, it has different dimensions like functionality, performance, security, usability, and maintainability. I could have framed it better to reflect that. I appreciate your insight!
In software testing and quality engineering, I define metrics as quantifiable indicators that help evaluate the quality, efficiency, and effectiveness of both testing processes and the resulting software products. According to ISTQB CTFL Syllabus v4.0, test metrics help track progress, measure test effectiveness, and support decision-making in the test process.
In software testing, metrics can be broadly classified into three categories: process metrics, product metrics, and project metrics. This distinction helps structure discussions around the specific purpose of each metric. Process metrics are used to assess the efficiency and effectiveness of testing activities by analyzing aspects such as the rate of defect detection or the progress of test execution. Product metrics focus on evaluating the softwareās quality characteristics, for example, by measuring the number of defects in relation to the codebase size. Project metrics, on the other hand, provide insights into the overall progress and stability of a project, including factors like variations in test effort or adherence to planned schedules.
However, it is essential to interpret metrics in context - numbers alone donāt define anything but serve as indicators that need analysis and correlation with business objectives.
What metrics do you find most valuable in your daily QA business?
āsoftware quality is not just one thingā
When doing software testing Iām not considering only the software quality.
Hereās some quality demands weāve been made aware of as a software development team by some stakeholders: increase revenue for a specific category of users, less customer support calls/tickets on specific topics, branding and reputation care, decrease gap against closest competitors, increase customer retention and satisfaction scores, adopt industry certifications and compliance for marketing or legal purpose.
The software quality isnāt in a bubble by itself, itās part of a product, business, company.