Hi everyone, at work we’re looking at rewriting job role definitions including for Software Testers.
What do other companys use as Key Performance Indicators for Software testers.
2 Likes
IMO KPIs are to be approached with a lot of caution. Its a minefield. People will intentionally or not, relentlessly work to fill exactly those measurements. Have a gander at recent articles about CD Projekt Red and outsourced QA who were working on Cyberpunk 2077. That outsource company set KPIs on their QA that were long ago established as completely counterproductive: number of defect reports per day. What they got were low pri junk defects filling up the KPI in favor over any actual deep rooted defects that took time to uncover and articulate.
So…be cautious and test your KPI as if you were trying to game that system.
8 Likes
Luckily I have none. I do my job as good as I can.
How I can help my team, company and our product is way more complex then to express in KPIs.
2 Likes
Similar to Michael’s view I have also found KPI’s for individuals very harmful.
What I’d recommend is first take a step back really consider what you are using the KPI’s for.
For example
Using a scoring system to determine salary increases or how top divide a financial incentive to the team. This could be stated out loud or perhaps be a more subtle use and agenda. It can also be used as justification to release people deemed as poor performers. This use of KPI’s is likely the most dangerous and will lead to team dysfunction in most cases.
Using KPI’s to determine promotions, this falls into the same category as the above.
Avoid adding any incentives or demotivators to individual KPI’s if you can. I’ve had managers believe changing the target meant the results would change by magic, all it did was people gamed results and yep magical change that was not real.
If the goal is helping individuals improve, this can work well however if its generic it will fail, if there is no action plan for change and support for individuals to change it will fail.
What worked was a bit of self assessment, individuals chose their own KPI’s ideally aligned and guided by company or and team level OKR’s or KPI’s and discussed with a coach where they are currently, where they would like to get to and how the KPI could be a loose measure of success. The coach would them work with them on an actual plan, often including training, experiments and use of tools to achieve better performance in their target improvement areas.
That is the only one I have ever come across at individual level that was of value, I’ve been involved with a number of these that actively did harm and the usual outcome was one person got a great score to show the team its possible, one got a poor score and often removed from team as a warning to others and most other people got slightly above average, you are doing okay but could do better.
Team level KPI’s can work better though particularly if they are aligned to company goals, I’ve been involved in OKR models that start with company and then team choose their own OKR’s guided by the company, the most important thing though is that the team is empowered and supported to change.
The goals for me are generally about helping people improve, if they are not its more likely that it’s me who is organizing the KPI’s that is failing and not the individual.
3 Likes
Here is something I think might be useful: Key Performance Indicators (KPIs) in Quality Assurance: Measuring Excellence in Software Quality
Notice the focus is on KPIs about the quality of the product, the process and the effectiveness of quality activities; and not on the performance of individuals. Its asking organizational questions instead - are we covering adequately with tests? are we seeing more defects in production? are defects being created at an increasing rate?
4 Likes
Not a fan of metrics. I always feel that people will gamify metrics (intentionally or not).
Also, how do you measure the overall individual’s contribution that are not apparent via KPIs?
Once you defined a set of KPIs, I’m sure people will focus on those and ignore everything else.
1 Like
I’ve used simple coverage metrics as KPIs for automated testing. I currently use completion percentages for phone calls as my current KPIs around the quality of our software product.
Don’t know that I’ve heard KPIs being attributed to individuals. Individual groups or business yes, not testers specifically because our work is so intertwined with everyone else’s.
1 Like
We seem to negotiate them all of the time wherever I work. Which is a thing that has made me bore of them. Things are always “going well” as long as we all agree they are going well even though inside we know something is not right. We always ourselves know just before we are about to get ill , before anyone else can spot that we are about to book a day off. Performance indicators are not an indicator of inner health at all. Either I’m doing my job, or you fire me, or the company goes bankrupt because the boss did not spot the rot. In some places KPI’s are like a chicken putting lipstick on the pig, and we all know who the chicken and who the pig is in this story by now @lgibbs . Teams work best when managers do the work of reducing friction, and no amount of KPI writing will make your personal quarterly goals get met.
Product KPI is less contentious. We have a target of 90% automation pass rate, before release for example. But until managers actually look at the use-case coverage in detail, then things like how hard the balance between manual and automation is to keep progressing without silly metrics, they become disincentives to automate deeply. Even manual tests that fail can be excused as with things like “oh we are not going to fix that until the XYZ release”. So any metrics are really targets, and I like targets like shorter closedown times for releases, but if “rushed” then quality seems to go down.
A good KPI is release cadence and failure-recall or hotfix-required frequency. And even though we do track release failures, very little actually gets done to address the root cause of release failures/respins in the end. For me, I’d really like an objective metric that I control, and I’m just not finding it either.
2 Likes
I liked Keith Klein’s approach to evaluating test managers. Three criteria: honesty, integrity, and accountability.
I’d think these should work well for testers as well, due to many companies not having test managers and the testers own the testing work they do.
Other things that could work well: involvement, communication, professional development, peer/colleagues review.
2 Likes
I would recommend reading the section on Evaluation of performance, merit rating or annual review in Out of the Crisis by W. Edwards Deming(1986, p101). He says “it nourishes short-term performance, annihilates long-term planning, builds fear, demolishes team-work, nourishes rivalry and politics”.
The Leader’s Handbook by Peter R. Scholtes is also worth reading as it gives an alternative approach
3 Likes
So I am the manager of a QA team and have done this for a few years now. My approach to KPIs for testers depends on a few things:
- Who is asking for them?
- How are they going to be used?
- How is collecting them going to change how the team works?
If you are a manager and you want to fire someone you can find a KPI that makes them look bad. If you want to reward someone you can make them look great with a different metric. Telling them that you are collecting any metric or worse making them report a metric is going to drive actual productivity down.
The only times I have had success with KPIs on an individual level was when I got my leadership team to separate it from performance / compensation and tie it to resourcing and recruiting. The point of tracking metrics then shifts from “let’s find the bottom 10% to cut” to “we have a gap in mobile testing on this team, lets hire for that”.
The way I tell if a member of the team is successful is if their opinion of value to the team matches what the other members think. If there is a gap I tend to find there is a communication gap one way or another and resolving that makes under performing testers much more apparent. I mean at the end of the day if a manager can’t tell if you are a good fit or not the manager is probably the one who should go.
To specifically answer your question of what other companies use, I have seen all of these used or attempted to be used as KPIs and did not find any of them reliable indicators:
Code coverage contribution
Test success rate (as a measure of flakiness)
Test plans written
Tests executed
Escaped defects on areas tested
Bugs found during a sprint
“Test case effectiveness” (This was never defined well enough to give a value to)
Time between a bug being introduced and being found
5 Likes
Welcome to the MOT community Sean. This is awesome, no honestly really it is good to hear well reasoned cases when KPI’s are useful for a change without gross bias language in the reasoning.
I always used to believe that things like your “escaped defects on areas tested”, although hard to quantify, was still a better metric by far than stupid things like code coverage, number of tests or flakiness. But at a high level one can see how such a belief can lead to a…
Lets test fewer areas so we get more time per area we do manage to test, mentality. Is that what you are hinting at there @sean_t ?
Nice. yes you are showing exactly why KPI measurements on an individual are almost always a poor choice. They are inherently biased to create individual work that games those KPIs, whether by intent or subconscious bias.
Systemic KPIs are useful, IMO. Which you also note. Where are the defects coming from? how long does it take to find and fix them? Which areas of product are seeing an increasing or decreasing trend in defects. Why? always the “why”.
1 Like
@conrad.braam Thanks for the welcome.
Lets test fewer areas so we get more time per area we do manage to test, mentality.
Yes that is the kind of thing I have seen when you expose low level metrics to high levels in the company. It never results in “We should get a bigger test team or give them the tools to hit the goals.” It almost always leads towards the decisions that will make a metric look better while taking on a bigger risk in an area that is currently stable or well tested.