Does better testing always lead to better software? This article, “Four reasons to stop testing: Finding the right balance in software quality,” explores the places in which you might reasonably scale back your testing efforts, while keeping an eye on the pitfalls and risks of holes in your test coverage. In the end, it is always a delicate balance between cost and benefit, as well as a tradeoff between the short and long term.
What You’ll Learn:
Which types of risks are mitigated by which type of tests
How future plans for development impact current testing needs
Where arguments against extensive testing are valid
What risks you have to accept when saving on testing costs
After reading, share your thoughts:
What criteria does your team use to decide when you have tested enough?
Have you experienced situations where reducing testing seemed reasonable? How did you decide?
What criteria does your team use to decide when you have tested enough? we say we’ve tested enough when test activity has reached the Testing Saturation Point — where each successive test run gleans marginally less significant info, that testing beyond this point seems repetitively non-significant.
Have you experienced situations where reducing testing seemed reasonable? How did you decide? none. It does seem reasonable to reduce generation of test evidences, as those types of test activities can be done post-release– a ‘Test-Drag’ if you will, slowing down the test team from pursuing new tests, in favor of completing evidence for future reference. But reducing the test-level applied? never — because it will only lower product acceptance credence; so reducing applied-test-levels is never recommended. We may test some critical criteria first, then release with a caveat (begrudgingly, as the product owner/s may want) acknowledging the inherent low acceptance credence — but we will definitely apply continuous exploration to the ‘product-under-test’ in Live, to increase the acceptance credence of the released product.
We call testing “done” when every critical user journey works, our risk matrix shows the risky bits covered, and we stop finding serious bugs after two runs.
For small UI tweaks or urgent hot-fixes, we cut the suite to a quick smoke test around the change.
Delivery stays fast and we have yet to see post-release surprises, so the trade-off feels right.
The problem we face in our team is when to say enough tests have been applied. This is usually not quantified by the amount of test cases applied but set upon certain key risk-based criteria: whether all critical functionalities are covered; issues with high impact are fixed; and the major, most common user flows are stable. Then remaining risks versus timeline, versus business priorities, are weighted.
Scaling-down on testing made sense in given situations: features being rather temporary, low-impact, or just little changes that were well-isolated. Here, based on targeted testing, automation coverage, and historical defect trends, we decided it was simply not worth spending more time and money on exhaustive testing. It’s always a trade-off between quick delivery and the quality our users expect.