My company is starting to look at how we measure and record AI usage for development and testing work, they want us to record something tangible that they can report on each month. Is anyone measuring time saved/quality improvements by AI. How are you doing it? Any ideas welcome.
Quite simply I stick to DORA metrics as at the end of the day, success is measured in outcomes. We adapted them slightly to meet our needs:
- Planned deployment frequency
- Unplanned deployment frequency
- Change Failure Rate (or for us frequency we are fixing production issues)
- Time to Restore Service
- Regression Testing Time taken (not a DORA metric, but one we want to measure)
- Rate of feature enhancement (not a DORA metric again, but tracking our frequency of deploying new features)
The bottom line is, is your new way of working improving outcomes? Those are the metrics that matter.
In our case, the clearest impact has been QA capacity.
We use AI to read developer-provided specs and code changes, then help design and execute the relevant tests. The useful output is not just generated test cases, but a go/no-go style report: what was checked, what risks were covered, what was not covered, and whether the change looks safe to proceed.
So I would not measure AI by prompts used or tests generated. I’d measure whether the QA team can assess more changes with the same people, while still producing useful evidence for release decisions.
The key metrics for us are: time from dev handoff to QA feedback, number of changes assessed per cycle, quality of the evidence, issues caught before release, and how much human review is still needed.
First I’d use the metrics I have like number of bugs by KLOC, time to deliver, recurrence of bugs… and them, according to the results I’d implement new metrics. I think using existent metrics first is a good approach, because you are able to compare the before and after IA. Using fresh new metrics you don’t have the a baseline.
The ROI of AI-assisted Software Development report released recently (by the DORA people) is also useful for how to tackle this, as it explicitly includes the cost dimension.
Thanks for the reply, Gary, that really helps. I wasn’t familiar with DORA metrics before, but they make a lot of sense. It makes sense to measure whether AI is actually improving outcomes rather than just tracking “AI usage” for the sake of it.
I guess stakeholders will look time saved in development and testing, because those are the easiest numbers to report. But I’d really like to shift the narrative toward quality improvements.
I really like your point about tracking the frequency of production fixes. It’s a simple, outcome‑focused way to show whether quality is improving over time.
I’m not sure how we can do this, if we want to compare AI‑assisted work with non‑AI‑assisted work, we can only do that by looking at historical baselines. Without that, it’s impossible to say whether AI is genuinely improving things or just changing the shape of the work.
Thank you for your reply, I’m building up a really useful list of metrics I hadn’t considered before. I’ve actually been recording the time from dev handoff to QA for the last six months, so it’ll be interesting to see whether that improves as we start introducing AI into the process over the next six months.
I haven’t been tracking many of the other metrics you mentioned, it reinforces something I’m starting to realise: if we want to compare AI‑assisted vs non‑AI‑assisted work, the only fair way is to measure against our historical baselines. Otherwise we’re just guessing.
That said, it might also make sense to start capturing these metrics now and then see how future AI improvements shift them over time.
It’s good point, it’s difficult to find out anything meaningful about what AI is giving you, without having the pre-AI data.
Interesting, thank you for that, I’ll take a look at that report.
If your stakeholders have visibility of time saved using AI, what does that prove? I have one team that believed they were more productive as they knew they were producing so much more code using AI, than previously. But our deployment cadence slowed. Because they felt they could do more, so changed more, which had a wider impact and those extra changes were untracked. So they broke more, had to back track to understand what they had changed.
For me, outcomes are the only measure. The measures highlight the symptoms, the cause then has to be investigated - which is how we established the above.
Your starting point is going to make a big difference.
What AI has been bring to light is that a lot of teams for a reason I cannot explain here seem to have been doing a lot of boring repetitive activities, have persevered for years with flakey automation rather than addressing it, spending a lot of time on reporting or very basically highly inefficiently doing activities that have always been much better allocated to machines.
The good news is if that’s the starting point you are onto a big winner in terms of showing tangible improvement, though to be fair you could likely get the same benefits just by stopping doing them. Perhaps I am being to harsh I am seeing a lot of this AI can generate test cases revelations. The big downside apart from the obvious is that you may find developers can absorb your value with AI completely.
It gets a bit harder when you are already 10x compared to the above, I’ll often test four products a day sometimes more and cover ten developers worth code as standard. The advantages of AI are definitely there but perhaps a bit more tacit so harder to put into words.
One key one for me is that as developer and feature throughput speeds up that I am able to maintain or increase the level of coverage. So far I am but I have not clear stats on this so could be many reasons. This is likely to be the key measure
I have some clear winners though. Getting basic automation up and running with short efficient health coverage of main feature flows is much quicker. We may even have not done this before based on ROI, now its straight forward tangible benefit.
Debugging with source code access. I had an obscure bug last week that the root cause was very specifically the user had accessibility clicker/switches turned on their device, not general accessibility settings that was starting point but very specific. Maybe 20 mins with tools to find this, I’ll stick my hand up here, I may not have found this without the tools. This is tangible both from speed perspective and finding root causes or issues we may miss otherwise.
Headcount will likely come up for larger teams, are less testers required.
What more has it empowered you to do, expand your risk coverage, there are good tools on security and accessibility that you may have lacked expertise before. Can testers take a small step into coding, not only efficient root cause but doing fairly safe bug fixing - I suspect we will drift more into build activities in general.
Data creation, vibe code very specific tool creation to help with a specific risk investigation, reporting, research.
I’ll skip the idea of full on agentic here though management will like that, I am still at the point just because we can does not mean we should and also questioning why should I give robots the fun testing stuff I love.
Be wary of rushing to put yourself out of a job, not only from a personal point of view but there could be bigger problems ahead by stepping to far from testing decisions.
Context will be a big factor, I’m solo small to medium mobile and web apps and not for example working on massive, complex banking systems.
One more example from yesterday. About 18 months ago we spent investigating patrol automation for mobile flutter applications, we spend about two months on it and got a decent UI layer health check up and running but it had a number of issues that we abandoned it as the roi was not there and to be fair it was still a bit flakey. We could have persevered and sorted that out but opted against it as we have those risks covered fairly efficiently elsewhere.
I have a hypothesis that AI changes that ROI equation, the code and tests were still there, I’m on a new laptop and the tool version has stepped up a number of iterations and has a lot of setup dependencies. It could have been a daunting task to revive it, with AI I pretty much one shotted get it back up and running.
It’s still a hypothesis as there will be CI aspects I need to investigate but that one shot updating and getting things running as before just makes these sort of investigations of new tools much quicker.