We are starting to get asked a lot about the return on investment for our QA dept. The questions are coming from business heads, not IT internal heads, etc…
Does your IT dept use any sort of metrics to show how much value your QA team brings to the company?
I know assigning KPI’s and metrics to measure a department can be a slippery slope, so just curious as to what others are doing that might be useful in communicating ROI to non-tech folks.
This is always a tricky position to be placed as you cannot truly measure the impact of QA without simultaneously releasing the same code both tested and untested.
I guess it depends what you are building and who your customer is, but a simplistic model to use might be, if you have a well defined priority/severity system for your bug management, you can provide a count of bugs found by QA above a certain threshold for any given release, highlighting those where the business would have been negatively impacted had a customer discovered them instead. Combine this with support-raised bugs (hopefully not too many) to determine the ratio, which would hopefully be a positive view in favour of QA.
Business heads can then determine if the benefit to product integrity and reputation is worth the investment.
I agree with @testingtesting1231; this is a problematic metric. Asking for the ROI of testing is like asking for the ROI on insurance. For example, if there are no defects in production, is the return zero? Maybe you measure customer happiness (NPS), contract renewals, new customers, etc. The first question is how you measure the return.
I would take a step back and try to understand the problem the business team is trying to solve. Then, maybe you can get to a metric that meets their requirement. Maybe try and move them towards an efficiency or effectiveness figure.
We discussed this topic in another thread only a few weeks ago, so I suggest you look for that. In my view (and that of any other reasonable person) is that there are no valid software testing metrics and that you should refuse to provide them. For me, this is non-negotiable - I simply won’t do it, and I don’t care if the idiots who ask the question don’t like it.
To pick up on Ken’s comments, the factors that management really care about are unmeasurable. What is the value of something that didn’t happen, such as finding a bug that would have damaged the company’s reputation? How do you separate your contribution to contract renewals from that of other departments such as development and after-sales support? The sales department would probably want at least some of the credit for new sales -they weren’t all the result of your great testing.
There’s an old saying, “Half the money we spend on marketing is wasted - we just don’t know which half.” The same applies to testing. And if you’re getting questions like this, I would say that at least half the money spent on business heads is being wasted. You might want to ask how they measure their own ROI.
Tricky Tricky Tricky!
ROI is a tricky concept in testing.
Any popular metric from the web that you apply, members will gamify that out.
However, some stuff can be helpful especially if you setup the right way to measure and infer it.
Here is something I wrote last month while thinking on this topic: Testing & Quality Metrics – Pitfalls to Avoid - Rahul's Testing Titbits
Also, the real “return” is Quality which cannot be measured very easily again!
I agree. How would you measure the monetary value of (higher) customer satisfaction (eg when software doesn’t crash anymore)?
Hard to measure.
Customer feedback, Feedback from Sales & Marketing folks, Feedback from Internal Customers (if any)…
Also, Quality is not always about Testing quality.
Ex: Product owner may decide to go ahead with a release having some bugs.
Or, Testing was hurried and wasn’t given appropriate time or the testing team is low on strength
Or, something else…
There are a lot of discussions on the harm of these sort of metrics, I’ve added quite a bit to these discussions before based on real dysfunctional experience with them, but I wont repeat that here.
I will though pick up on the business head stakeholder aspect though.
Quality itself is very much at the core of business value often to the extent that quality is often measured directly in terms of business value.
That is the discussion to have, ask them to talk through all the business values of the product, ask them to talk about that in terms of quality, how a low quality product could impact that business value.
These discussions should be happening regardless but bring it up front and center, if they can give a clear definition of this, this will in turn guide the actions of your QA department and you can then talk about how this contributes to those quality goals.
It is a fairly hard level of thinking and not something they should pass on to QA team to own, it needs to come from business focused people.
They will though no doubt struggle to put metrics on it, any quantitative metrics will be abstracted to certain level, often could be gamed and could be harmful so used with great care. More often than not qualitative measures are much more appropriate but they are harder to measure.
Great article, Rahul! I also think customer feedback is a great way to gauge the value of quality and testing efforts, and I also tend to lean more towards trends than absolute numbers.
Something that comes to mind is context and observability. I think it’s interesting to pull key data and see whether certain changes that were made have a visible impact in that data. Or the reverse - trying to come up with potential reasons for the trends you’re seeing. For example, number of bugs reported going up when new testers are onboarded, or customer satisfaction scores increasing after an improvement suggested by a tester has been implemented and released.
Building on Morgan’s ideas, you could also hone in on the issues found by testers that would have the biggest negative impact, both on users and the business. “Nightmare” scenarios or headlines may help the business heads to better understand the impact the QA team is having. Find out what’s important to them and find a way to relate that to the work of the QA team.
These are all well thought out suggestions. I appreciate the feedback.
In my 12 years of experience in QA I have seen multiple methods “attempt” to ascertain the ROI of Quality departments in software. So I was curious as to others experience.
I have had dept heads describe us as a “nice to have” all the way up to “essential” to the dev process.
We are building a team to a degree (or rebuilding) so the question comes up as we have recently spent a few dollars on QA. It’s natural to have a spotlight in that instance.
Thanks for the input.
Usually it’s Cost and Reputation Metrics like:
The most important part is to understand what business track and try to correlate to their metrics then it’s understandable for business stakeholders
Mutation Coverage <3
For those who don’t know what Mutation Coverage is, it comes from mutation testing, a way to test if you’ve written good unit tests because code coverage is bullsh*t 
Meaning if you have good mutation coverage, you’ve got some good unit tests already. And it speeds up our development also. It shows us the “doing things right the first time”. We used to write unit tests, still find a lot of bugs because their were unit tests missing due to them not being testers. Which now helped in the quality aspect and less bugs go to the testing environment, since they are caught during mutation testing & unit testing.
We probably lack a lot more ROI reporting now I think of it …
I recently made a post about UI Automation and if it gives an RIO, interesting to see if someone comes up with one
Interesting segue into the next question our leadership is being asked thus asking for….we are also being asked as to what code coverage we currently have in our UI automation in the same breath as to how we can show our “business” folks what ROI our quality team brings.
I have never thought code coverage mattered for UI automation. I always thought this was a marker utilized for unit testing or AAT’s.
Now, being able to talk to the feature coverage is more realistic, but they don’t necessarily want that.
I’m not talking about coverage for UI automation, I was pointing out ROI for UI automation 
Terms like QA Department and QA Team are troubling. They represent silos that impede delivery. As for the ROI questions you’re getting, that suggests QA layoffs are incoming.
Both QA and agile coaches have the same problem - and the impact on ACs over the last few months has been quite negative. Unlike developers, QAs and ACs suffer from cycles of high and low employment. In an economic downturn, business leaders will start looking at what can be cut.
The problem for QA/AC is that they both find it difficult to justify their value. It is not an impossible ask, but certainly difficult. The reason is that success is measured in the absence of things, rather than their presence. This is something that comes up regularly on Agile LinkedIn for coaches and scrum masters.
I think the key is to know your audience. Providing metrics that are of value to QA wont have any value to business leaders. So what is the value of QA to the business? The unfortunate position is that it involves risk reduction, rather than something more tangible. Many leaders, especially when money is tight, will absorb that risk rather than continue with costs that the business simply cannot afford.
I see this as a discussion about risk management.
That’s true but also could mean Squads or Guilds
It’s easier to say department then Squads since people will ask more questions about squads.
Nevertheless if it is a QA Department… they do still exist in silo’s that’s correct.
I wouldn’t say layoffs are coming but sometimes they are working on audits or certifications for the company and they require certain reports… I know… 