When it comes to quantification techniques, there’s a lot talked about code quality, but a lot less about overall customer quality, other than feedback surveys, and who likes doing those?
I learnt about Evo (Evolutionary Value Delivery) years ago as a way to quantify quality. Some say it is one of the original Agile methodologies. I’ve seen it used as a way to translate business language and requirements into measurable outcomes.
Example:
The sign-in process should be ‘easy’ and ‘intuitive’.
How do you measure ‘easy’ or ‘intuitive’?
Put measurable values on them. E.g. It will be very easy and intuitive if someone who has never seen the software can sign in, in under 15 seconds. Easy if under 30 seconds. Too hard if more than 30 seconds.
Now it is measurable.
Yes I agree, ticket counts or bugs found doesn’t work as a meaningful measure of performance. I like the idea of improving the team’s ability to test with confidence, it feels like a much more valuable direction, your examples are giving me some good ideas for setting goals.
I’m aiming to move further into automation, so I’m sure I’ll be referring back to these suggestions as I progress. At my company we still follow a fairly traditional waterfall process, which works for the size of our projects, though I’ve wondered whether adopting some agile practices might help. Either way, improving the speed and quality of feedback to developers seems like a useful measurable goal. Better documentation is a great idea, with the opportunity of using LLMs in the future, having solid documentation seems like a must.
I’ll look into Tuskr and Qase then, I’m not using a test management tool at the moment, it’s definitely something I need to explore. I imagine I’ll be posting more questions about that at some point.
I really like the idea of framing goals as user stories. It’s so easy to lose sight of why a goal exists, so focusing on the story first and then deciding how I’ll know I’ve achieved it, feels like a great way to stay aligned with what actually matters.
Thank you as well for the reminder that quality is much broader than automation. I agree. Some of the most valuable contributions I make come from noticing an angle we haven’t explored yet, or drawing on past experience of what’s gone wrong before. That instinctive side of testing is still important, even as I’m learning the value of automation.
I really appreciate you taking the time to share this. It’s given me a much clearer sense of direction for shaping my goals.
Thank you for pointing me toward Capers Jones. It’s useful for me to broaden my awareness of what’s out there, and I’ll keep your perspective on the limitations of his approach in mind as I read.
I also appreciate the insight into context‑driven testing. One of the goals I’d drafted was to learn about “best practices”, but you’ve made me realise it’s more valuable to focus on which practices actually work best for our context.
The links and references you shared are genuinely interesting, there’s a lot to explore. I’ll definitely be digging deeper into session‑based testing and the RST methodology.
To avoid any misunderstanding, I mentioned Capers Jones as the perfect case study of what you should NOT do. Anyone with a reasonable understanding of mathematics and statistics will be able to confirm the absurdity of his work for themselves.
One of his favourites is to produce pages of calculations that don’t match his measurements. He then “fixes” this by arbitrarily applying a fudge factor on the basis that this is what you need to do in this particular context. However, the fudge factor is magicked out of nowhere and is different for every context. It’s obvious that the calculations are simply wrong and have no predictive capability, but Jones refuses to acknowledge this.
He has also stated that his methodology assumes that the organisation is CMMI (Capability Maturity Model Integration) level 5, which almost no organisations are. This is just deflection, presumably to discredit anyone who doesn’t work for such an organisation. However, it’s irrelevant because his maths is wrong because it’s based on assumptions that are not true in any context.
Thanks, your point about measurable not needing to mean “countable” really resonated. I’ve ended up shaping my goals around investigation and improvement rather than numbers.
I’ve set myself things like doing a feasibility study on automating some of our current manual tasks, and another around exploring how AI could support what we’re doing.
Really appreciate the perspective, it helped me frame things in a much more practical way.
In my organization, KRAs are being set by the organization itself, and our goal is heavily influenced by KRA’s.
For e.g. time estimation which is one of the crucial KRA, so obviously our goal is to complete the ticket within time and if there is any issue or blocker, raise it asap and then coordinate with dev/PMs to get it resolved.
Other goal is to keep improving the test strategy to enhance test coverage.
The goal is subjective topic which varies person to person, but being in organization while setting the goal, KRA should be considered, else self golas can vary from organization which can somewhere hamper the progress within the organization.
Wow, that’s opened up a whole world of new things for me to look into. I’d never come across EVO or Planguage before, so thank you for pointing me in that direction. It looks like Competitive Engineering by Tom Gilb is considered the foundational work in value‑driven engineering, so that’s now firmly on my reading list. The whole idea of actually quantifying qualities, is really interesting. I’m going to dig into this a bit more.
Thanks, this is a really useful way of looking at it. I think I need to take some time to understand what my line manager is aiming for and what the company is trying to achieve overall. I have quite a lot of freedom in how I approach testing, so it makes sense to shape my goals around that bigger picture. I recently started tracking how long it takes me to get back to developers after they send work my way. I thought it might make a neat goal, but doing it has been more interesting than I expected. It has helped me spot which work items tend to sit untouched and why. Most of the time it comes down to not having enough information about what changed, so I have been putting effort into tackling those gaps early.
It feels like I am finding good ways to improve quality in general, but I agree that proper goal setting needs to line up with what the company is trying to do. This has given me a clearer direction.
Thanks, that’s really interesting to hear. I’ve mostly worked in smaller organisations where things like KRAs weren’t really formalised, so they were never a big part of how goals were set. Hearing how they shape expectations in your company makes me think I should look into whether we have anything similar, or whether it’s something worth suggesting. It would definitely help me align my goals with what the organisation actually cares about.
At the moment I’m still trying to get a clearer picture of what my line manager and the wider company are aiming for, so having something like KRAs in place would make that much easier. It feels like the right direction for making sure the goals I set actually support the team and the business rather than just being things I happen to find useful.
One thing worth naming directly is the pressure to quantify quality often comes from outside the QA team, not within it. Managers want dashboards, review cycles need numbers, and so testers end up reverse-engineering goals to fit a format that was never really designed for this kind of work.
So before asking “how do I measure this?” it’s worth asking “who actually needs this measurement and why?” Sometimes the honest answer is that the goal exists to satisfy a process, not to improve anything. And once you see that, you can have a more useful conversation with your manager about what would actually demonstrate impact.
For Jonathan specifically → you’re in a small team, starting from scratch, which is actually an advantage. You get to define what good looks like before anyone else does. Rather than fitting your goals into a metrics framework that doesn’t quite fit, use this window to propose goals around building the foundations: a lightweight test strategy, a way of classifying where bugs are coming from, a feedback loop that didn’t exist before. Those things have visible before-and-after states, which makes them far easier to defend in a review than any bug count ever would be.
@checkout_champion nobody in the world cares about ur goals as much as you do…dont let anyone dictate your future , even if u have achieved all ur goals u have set in ur company, it is not necessary these companies will reward you…but its always be proactive rather than passive on this…think about what boosts ur energy , things that make u feel stimulated and invigorated and work in that direction… but if ur looking for a raise, then its necessary u set goals , in the direction that company finds value in it , n it is not necessary that ur personal goals and things that company finds it valuable align
Firstly, Quality is difficult to quantify directly, so in SQM we focus on indicators that reflect quality across prevention, detection, and customer impact.
Key Areas Where QA Goals Can Be Measured
Defect-Based Metrics (Defect Detection)
These indicate how effectively testing identifies issues and Reduce production defect leakage from 8% to <3% per release.
Examples
Defect leakage to production
Defect density (defects per feature/story point)
Severity distribution of defects or Issues
Reopen rate of defects
Test Metrics
Measure how well test coverage identifies issues in order to Achieve 95% requirement traceability coverage in lower environment testing.
Examples
Requirement coverage (%)
Automation coverage
Defect detection percentage (DDP)
Test Automation & Effectiveness Metrics
Especially important in Agile and DevOps environments should Increase regression automation from 40% to 70% for every releases.
Examples
% regression automated
Automation execution time reduction
CI/CD pipeline pass rate
Release Quality Metrics
Focus on business outcomes that improves to lower the critical production incidents by 55% over 3 to 4 releases.
Examples
Production incident rate
Customer-reported defects
Mean time to detect (MTTD)
Mean time to resolve (MTTR)
Process Quality Metrics
Measures improvement in engineering practices and Ensures 90% of defects are detected before UAT.
Examples
Requirement clarity index
Defect removal efficiency (DRE)
% shift-left testing
Balanced QA Scorecard Approach
Strong QA leaders combine multiple metrics instead of a single number.
Thank you, this is really useful to me in my current situation. I’m in a phase where all of this matters a lot. Your point about analysing why certain measurements exist is such an important reminder. At my last (much smaller) company it was easy to see the whole picture and understand the purpose behind every process. Moving into a bigger organisation with established structures and goal‑setting frameworks, I can already see how easy it would be to forget to question whether a metric is actually meaningful. It’s encouraging to think of this stage as an opportunity rather than a constraint. Being part of a small, growing QA function means I really do get to help define what “good” looks like. I love the idea of focusing on foundations and using clear before‑and‑after states as the basis for goals, it feels far more honest and far more defensible than trying to force numbers onto work that isn’t naturally numerical. I really appreciate your message. It’s given me a clearer direction and a bit of excitement about shaping things from the ground up.
Interesting, this is a perspective I hadn’t really considered before. I’ve been so focused on defining “good QA goals” that I didn’t step back and think about the bigger picture for my own career and motivation.
It makes sense to choose goals that energise me and help me grow, even if they don’t perfectly align with what the company values. The company still benefits when I’m motivated and developing in meaningful ways, even if the goals originate from my personal direction rather than theirs.
I suppose the challenge is finding that balance, choosing goals that support my long‑term development, feel meaningful day to day and still show value to the business. Thank you, that’s helped me re-think how I’ll set goals in the future.
Thanks so much for taking the time to write this, it’s genuinely eye‑opening.
I’ve spent most of my career in smaller teams where QA is important but not formalised, so I’ve never really seen quality broken down in this kind of structured, measurable way. Your explanation of SQM and the different categories of metrics really helped me understand how larger organisations approach quality.
It’s made me realise how much can actually be quantified when you look at prevention, detection, efficiency, and customer impact as separate dimensions rather than one vague concept. I can already see a few areas where I could frame my own goals more meaningfully, especially around early‑stage prevention and building automation capability.
I really appreciate you sharing this level of detail, it’s given me a lot to think about.
QA field experiences this issue as a frequent challenge.
My focus will shift away from “counts” which include bug numbers and test case totals to evaluating the actual effects of testing efforts. The testing efforts will focus on critical areas which need better test coverage to minimize escaped defects while stabilizing the automation system and decreasing feedback time.
You can create educational goals which involve both learning and your contributions to automation by developing a small automation suite which targets essential processes and enhancing current operational methods.
Measuring quality presents difficulties, but organizations can demonstrate their outcomes through improved confidence levels and reduced risks and increased team productivity.
I agree, counting bugs or test cases doesn’t really tell you anything useful. I like the idea of focusing on the effects of testing instead; it feels like a much more sensible way to think about goals that are actually meaningful.
Educational goals also seem worthwhile. Picking up skills and learning specific tools or techniques is always valuable, and I’ve included some MOT courses in my goals for that reason. It gives me something concrete and measurable to show that progress has genuinely been made.
Framing success in terms of reduced risk, improved confidence, and better team productivity definitely gives me a clearer way to think about impact. I guess the real challenge now is figuring out how to demonstrate those outcomes in practice.