How do you measure testing

Testing Thoughts
#1

The illusion of measuring testing.

Is testing a cost? Or is it an investment to gain some insights as part of the learning about what the product does and risks associated with it?

What price tag would you like to put on the insights?

Discuss :point_down:

1 Like
#2

If you’re in the business of developing software, and you do not test reasonably effectively, then you don’t stay in business. Consider Windows 10. Pretty awful, right? The update process is simply dreadful, often breaking important stuff. Yet it limps along and most of us use it to some degree. Now imagine how completely unusable it would be if Microsoft did no testing at all. Testing is an investment in your business, not a cost.

1 Like
#3

CFO asks CEO: “What happens if we invest in developing our people and they leave?”
CEO: “What happens if we don’t, and they stay?”
-Peter Baeklund

I think the same goes for testing.
‘What if we invest in testing and they don’t find bugs?’
‘What if we don’t invest in testing and there are loads of issues in production?’

What the pricetag is, depends on many kind of risks and costs. As always, the context matters.

1 Like
#4

Also known as “insurance”.

It always amazes me that people who wouldn’t think of not taking out assorted flavors of liability insurance in case of expensive lawsuit turn around and think they don’t need testers (or don’t need better or more testers) to protect their expensive intellectual property (their software) from bugs.

It may not be the best analogy, but it’s one that works on the folks that won’t do anything if they can’t put a price tag and an ROI on it. If an expensive lawsuit comes around because the software had critical bugs that were deployed and caused injuries, deaths or major financial issues, and the company didn’t have testers, that company would be in deep trouble. If they did have testers, they would stand a reasonable chance of arguing that the circumstances of the bug(s) were something that wasn’t detectable in the test environment.

2 Likes
#5

I agree with Kate. I’ve taken to (informally) promoting the ranking of bug severity on a scale of 1 to 10, where 1 is “everyone on the interwebs is laughing at our product” and 10 is “someone died”. In both cases, the follow-up question is “And how does that impact our bottom line?”

1 Like
#6

A quote from a speech in 1968 :
"
Gross National Product counts air pollution and cigarette advertising, and ambulances to clear our highways of carnage.

It counts special locks for our doors and the jails for the people who break them. It counts the destruction of the redwood and the loss of our natural wonder in chaotic sprawl.

Advertisement

It counts napalm and counts nuclear warheads and armored cars for the police to fight the riots in our cities. It counts Whitman’s rifle and Speck’s knife, and the television programs which glorify violence in order to sell toys to our children.

Yet the gross national product does not allow for the health of our children, the quality of their education or the joy of their play. It does not include the beauty of our poetry or the strength of our marriages, the intelligence of our public debate or the integrity of our public officials.

It measures neither our wit nor our courage, neither our wisdom nor our learning, neither our compassion nor our devotion to our country, it measures everything in short, except that which makes life worthwhile."

1 Like
#7

It’s because of things like the above that happen when you measure a more detailed thing than was asked for, I prefer black/white metrics. Almost to the point of trivializing, when a scale of 1-10 for severity looks nice on a graph, a scale of 1-3 for severity gets significantly quicker to vote on and assign.