What do you expect a test investment to achieve?
You’d need to know both what you’re trying to do specifically plus that it was your investment in testing that caused it rather than some other factor. Sometimes changing a system can yield results without the change being the entire factor. In one famous case in testing the effects on lighting on the efficacy of office workers the workers were split into two. One was given increased lighting and the other was given decreased lighting. The increased lighting group showed a statistically significant increase in performance… but that performance increase was almost matched by the other group. It wasn’t the lighting alone causing a difference but the fact that attention was being paid to their efforts. If you’re going through with it then you need to know what you’re measuring, how you’re measuring, when you’re measuring (Before and after? When will changes take effect?), but most of all… why.
Proper, rigorous, scientific research that provides values within a reasonable error margin is very, very hard to do and costs thousands of pounds for the smallest studies. A phase 2 clinical trial costs several years and around $20m, and that’s just to check if one drug has any effect at all at doing one thing - not if it’s better than anything else. So you also need to set expectations about what your investigations are capable of achieving. Maybe it’s not going to be pragmatic to expect that you can measure anything that will tell you want you need to know with a degree of accuracy. If your job is to make people feel better about that investment, or give reasons to one management level above you so they can answer difficult questions one level above that, then you can do that without too much cost or effort - if you look for a result you will find it. Collect a bunch or results, cut the negative-looking ones, and put it on a nice template. Knowing the result before you test is a scientific crime, but maybe you don’t actually need solid, reliable results. Ask yourself how accurate and true you want to be versus your desire to get a good outcome. You have a personal interest in a positive result and there’s no need to “prove” when all one needs to do is convince. Come to me with a measurement and I can find 10 ways to make you feel certain of your result and subsequent plan of action and 10 ways to make you doubt that you ever found anything at all.
If you still want something to examine you must consider what is measurable at all. If the goal is to ensure that client expectations are met then you should be sure you have those expectations. If you don’t fully understand and appreciate the expectations of every one of your clients then your goal is technically impossible to measure… or it’s poorly worded. You can’t measure a goal that’s written to be unmeasurable (unless the aim is to fake the results, in which case this is a classic way to pull that off - shoot for vague and you can achieve anything!). You can ask them, of course, before and after the investment, on a 1-5 scale using the exact same question wording and layout, to see if their opinions change. You’ll need to consider that the effect of introducing testers and test process changes will make a system worse before it gets better, with the disruption introduced to teams and working patterns. You’ll need to consider that it will take time to see any effect as the teams reform. You’ll need to consider how the people interested in these measurements will react - and they’ll react to support their own worldview. And after all that cost and effort you may wish you’d known why you bothered…
Consider if this is a political request. You’re given an impossible task - is that so you can lie to them so they can lie to someone else? Maybe they’re going to fire all the testers and want some “evidence” that testing is not a valid practice. I’m not saying that these things are the case, but they are vital contexts that completely change what one might do. If you bring this work to someone they can easily dismiss it as not accurate (for whatever reason) and do whatever they want AND convince themselves they gave it a fair shot because it’s someone else’s fault. The will of people will triumph over data in most situations. Remember that double-blind randomised controlled trials were invented to take personal influence, even subconscious influence, out of the equation as much as possible - the number of “critical defects” in a system can go down by me saying “well it’s technically got a workaround so is it reaaaaaally critical…?” over and over again. Don’t fall into the trap of thinking anyone’s exempt. If you find yourself tempted ask yourself what you’d genuinely do if you found that investing in testing showed no improvement whatsoever in quality, velocity, customer retention, employee retention, support call frequency, team happiness and bug report severity - would you accept the result and become a cabin-dwelling spoon whittler, or question the efficacy of the protocol? Do we decide to downsize our test team, or is it just the way the testers are working that’s causing these numbers? Can we fire a crowd of human beings based on our measurements? If not, what can we do based on our measurements? Whatever we wanted to do anyway? Because when faith in the numbers from an investigation done by someone who knows they are not a professional scientific researcher, and who knows little (enough to be dangerous) about practical metrology and applied inductive methodology (i.e. me), meets real world decisions it’ll be human conscience and consciousness that changes our reality, not some numbers I pulled out of patterns and complexity I don’t understand with meaning that I applied to them out of faith and arrogance rather than the understanding of a whole team of people who do it for a living. I keep in mind the humility required of testing some piece of software that won’t hurt anyone, so I should apply that to numbers that fuel business decisions that might.
So, with that in mind, what do you expect a test investment to achieve?
P.S. There are things that can be measured. I measured our cycle time and tester time when I was reducing a regression suite. The goal was to reduce cycle time and tester time by reducing a regression suite. That’s quite different from the question “is testing worth it?”. I’m trying to offer actionable advice these days, so… maybe ask why they suddenly want measurements for the benefit of the test investment. After all they’re not asking you to measure the benefit of the company’s investment in them. Either they’re making a business decision based on the cost-benefit of testers (e.g. someone important wants to restructure the teams and fire the testers), or who cares and why should you waste your time checking?
), I’m really skeptical about trying to have objective measurements that would actually be useful. Plus, the effort to find objective measurements makes it very easy to pick things that are easy to measure but not necessarily useful at all (consider all of the damn “SMART” goals that many of us need to come up with for personal development plans / reviews.