Ask Me Anything: Test Strategies

That is a trick question. A test strategy is a mental model. Not a document. :wink:

I can’t show you the examples of the Test Strategy documents, but I can describe them: I made a 50+ page document called “Master Test Plan (MTP)” describing everything about the test process: summary of project (copied form the project plan), management summary, deliverables, scope, responsibility assignment matrix, workflow, stakeholders, escalation paths, test organisation, stakeholders, conditions, test basis, quality gates, exit criteria, test types, findings process, planning and budget, infrastructure, test environments and 2 or 3 pages on PRA and Test Strategy.

Google “Test Strategy” and the majority of links you will find are like that: templates for documents full of fluff. Or vague and general statements about risks and a test approach.

Back to my bad examples: in that time (around 2002 - 2005) I wrote MTP with vague Test Strategies. It took loads of time and effort to make those documents and not many people would read it. Waste of my valuable time. Some parts of it were useful, but my team did not read the document because it was boring, full of fluff and it lacked what they wanted to know: what and how are we going to test? My document never became very specific on those topics. And, another big mistake: the document never got updated during the project. Once done, it was signed off and put in a drawer. I learned that testing is a team sport. You have to create a test strategy with you team. Get people involved by talking to them instead of sending them documents.

The PRA was a list of quality attributes with a simple calculation (possible impact x chance of failure) and based on the number we assigned a “risk class” (high, medium, low). And based on the risk class we assigned test intensity (heavy, medium or low) testing. This method is still taught to people. Look here to see what it looks like.

What went wrong is that we created a document upfront. It was fixed. While I now know that learning is a process, just like your test strategy: it is a process! It develops over time. We assigned test intensity to risk classes and we even decided upfront which test techniques we would use. Without really knowing what the software would look like. Dangerous assumptions. Now I know that it is important to develop the test strategy while we test. The more we know, the better we are able to think about potential problems. And the better we can decide what we need to do to find those problems.

I also created a PRA (product risks analysis) in a big workshop with lots of people. It took a lot of time and wasn’t valuable enough for the attendees it seemed. The lists we created were good and the discussions created insight. But I also learned that talking to people in smaller groups creates deeper discussions. And talking about risks and my testing over time going into more detail (you could call it evolutionary test design) helped me and the stakeholders to learn faster and create deeper understanding of what was really going on.

My old approach assumed “the higher the risk, the more important the test to be performed and the more test intensity we need.” Now I know that high risk does not necessarily mean that the tests also need to have more depth or a more formal test design technique should be applied. To find a diversity of problems, we need to use a diverse test strategy, which means a diverse set of approaches, techniques and tactics.


(source: Black Box Software Testing by Cem Kaner & James Bach)