Testers embedded in the teams. Developers own automation, but testers involved. Ratio 1 tester /6-8 devs.
Just going to drop this in - because using a ruler can often just end up covering important things in the bigger picture.
THis is hard because a lot of what a tester does influences how the team works and vice versa.
I would steer clear of things like bug counts and number of test cases written/executed sorta thing and instead look for:
- how is the team overall?
- how is the relationship between the tester and his/her team members?
- what skills have they been learning and how have they applied those skills?
Bug counts and number of test cases are excluded.
I know it is hard but asked to come up with measurement, and I want to be realistic and fair…
As with any measurements they are data, data to validate or invalidate a notion you have. So when you take about measurements you need to start with what you want to know. For instance, I want to know how much I should pay testers, so I need to know which of the testers that produces most value. What measurements can I put in place to measure how much value they produce? Or you might want to know if 1/6 is to much or to little, what measurement do you put in place to understand that better? So my recommendation is always to start there and figure out measurements later. Also drop the key, no such thing. Performance Indicators are key or not key totally depending on the question and thus should only exist as long as the question exists. And a more creative and constructive way to think about it is: Right now I am trying to improve this , but I will stop improving that as soon as it is good enough for something else to be my main concern. Thus I do not think it is helpful to search for the universal measurement, since that typically means, for it to be universal it also have to be weak. As in to abstract to provide good data for useful insights.
Let’s take bugs found as an easy thing to measure. Not all bugs are worth the same, so quantity is not enough to describe value. Also if you work in a team that you as a tester have a good collaboration with, so they will not produce many bugs for you to find. Or if you are a crappy tester that do not find the bugs that are there will both report few found bugs. Again to abstract.
So by removing the key, you can instead focus on harder to collect and more specific data to extract. Since it will be passing anyway. And by starting with the purpose it is also easier to convince others to do the work of collecting the data if they buy in to your purpose.
All that being said here are a few examples that have helped me:
- To help turning up or down test coverage we introduced a bug value so for every bug we found we added a monetary value of how much money we will lose if we have this bug in production. This helped the individual testers to understand the value of that part of the product and also what they should put there effort on, as well for us to scale up high value areas and down low value areas.
- To see if we should invest in a project to help with infrastructure we gather how large a portion of your time you spent on “productive work” like testing the software vs. “non-productive work” like setting up / troubleshooting environments, reporting bugs, reporting on metrics etc. (our concern was the environment) which allowed us to decide and scale a project to make it better, and also evaluate the improvement that effort had.
Both activities were transient in nature.
Incentivizing the raising of bug duplicates or variations, is merely a way to create more work. Evaluate your team based on how quickly they can execute test scripts, or can write test scripts, and then evaluate them on how quickly they verify defects that are fixed by devs. There is nothing wrong with giving a hypothetical “money-value” to bugs, but when you spend more than 5 minutes deciding if a bug is a P1,P2 or a P3, you have wasted time better spent getting it fixed, verified and released. Some of the best bugfixes don’t have Jira tickets, and get released with low cost and low risk.
After all, the time to raise, fix and release a fix is the only metric you should be using, and as such any things the team and individuals do that make it possible to achieve more regular releases is all the business cares about. Recovery time.