Ask Me Anything: Test Strategies

That is a trick question. A test strategy is a mental model. Not a document. :wink:

I can’t show you the examples of the Test Strategy documents, but I can describe them: I made a 50+ page document called “Master Test Plan (MTP)” describing everything about the test process: summary of project (copied form the project plan), management summary, deliverables, scope, responsibility assignment matrix, workflow, stakeholders, escalation paths, test organisation, stakeholders, conditions, test basis, quality gates, exit criteria, test types, findings process, planning and budget, infrastructure, test environments and 2 or 3 pages on PRA and Test Strategy.

Google “Test Strategy” and the majority of links you will find are like that: templates for documents full of fluff. Or vague and general statements about risks and a test approach.

Back to my bad examples: in that time (around 2002 - 2005) I wrote MTP with vague Test Strategies. It took loads of time and effort to make those documents and not many people would read it. Waste of my valuable time. Some parts of it were useful, but my team did not read the document because it was boring, full of fluff and it lacked what they wanted to know: what and how are we going to test? My document never became very specific on those topics. And, another big mistake: the document never got updated during the project. Once done, it was signed off and put in a drawer. I learned that testing is a team sport. You have to create a test strategy with you team. Get people involved by talking to them instead of sending them documents.

The PRA was a list of quality attributes with a simple calculation (possible impact x chance of failure) and based on the number we assigned a “risk class” (high, medium, low). And based on the risk class we assigned test intensity (heavy, medium or low) testing. This method is still taught to people. Look here to see what it looks like.

What went wrong is that we created a document upfront. It was fixed. While I now know that learning is a process, just like your test strategy: it is a process! It develops over time. We assigned test intensity to risk classes and we even decided upfront which test techniques we would use. Without really knowing what the software would look like. Dangerous assumptions. Now I know that it is important to develop the test strategy while we test. The more we know, the better we are able to think about potential problems. And the better we can decide what we need to do to find those problems.

I also created a PRA (product risks analysis) in a big workshop with lots of people. It took a lot of time and wasn’t valuable enough for the attendees it seemed. The lists we created were good and the discussions created insight. But I also learned that talking to people in smaller groups creates deeper discussions. And talking about risks and my testing over time going into more detail (you could call it evolutionary test design) helped me and the stakeholders to learn faster and create deeper understanding of what was really going on.

My old approach assumed “the higher the risk, the more important the test to be performed and the more test intensity we need.” Now I know that high risk does not necessarily mean that the tests also need to have more depth or a more formal test design technique should be applied. To find a diversity of problems, we need to use a diverse test strategy, which means a diverse set of approaches, techniques and tactics.

(source: Black Box Software Testing by Cem Kaner & James Bach)

I am not sure what you mean by the words “adhered to all levels”. Level could point to test level or level in the organisation. I assume that you mean: How do I make sure that everybody in the organisation is on the same page".

By talking to people about the testing! By talking to them about stuff they care about. By showing that you are helping them by delivering the information they care about (which is probably not details of the testing you do, but is a story about risks and quality). It is hard to satisfy different stakeholders, so talk to them about what they really need from you. Also see question 7.

I prefer to talk to people over sending documents, that is an interactive way of learning if my thoughts match the other persons thoughts. It may sound strange, but I meet a lot of people who think talking to others costs too much time or is hard to organise. They prefer to write stuff down. Which is okay too, but at one point we need to talk about it to gain deeper insight. Also: don’t wait too long to talk with others. Don’t try to be complete or perfect, build in fast feedback loops in building your test strategy.

I like to use models/visualisations of my thoughts to make conversation easier. I created workshops on thinking and working visualy with Jean-Paul Varwijk and later with Ruud Cox. Later I learned about storytelling and how to deal with complexity. I think the complexity of what we have to deal with (complexity of the products we are building but also the complexity of working with people: communication and understanding each other is hard!).

See question 15, question 32 and question 33.

This is a bit off topic, but I am happy to answer your question. That is, if I understand what you actually asking.

  1. Do you want to know how to become better in automation in testing?
  2. Do you want to know how to make automation in testing part of your test strategy?
  3. Or do you want to know how to improve the performance of your automated checks?

Ad 1) Have a look at these AMA by Richard Bradshaw or Mark Winteringham. Maybe read this post by Bas Dijkstra.

Ad 2) By collaborating with the whole team. The most important thing you can do is talk about the risks in the whole development lifecycle with your team. And discuss measures to deal with them. Start with why you want to automate. Talk about that first. Automation is not a solution on it self, it must solve a specific problem.

Talk to your developers about what you want to test and ask how they would do it. Ask for help form people who have experience in setting up automation to assist their testing in a way that solves your problem. Experiment and find a way to find out if your solution is successful. Talk about testability. Also have a look at question 26.

Ad 3) This is not a question I can answer without getting some context form you.

First by NOT thinking of a Test Strategy as an artefact. That is very helpful in understanding that there is a lot more going on than we write down. Tacit knowledge and the use of heuristics is very important in dealing with complexity.

Secondly, I do not think there is a true look of any document. That implies there is one way to deal with documentation. I think how to document and what to document highly depends on the situation.

Looking at documenting my test strategy, I like to keep them lean by adding visualisations. I keep them lean by not writing down everything and telling good stories about my work to the people I work with. I keep my documentation lean by using notepad more that MS Word so that I do not focus on format and lay-out too much. I keep documentation lean by taking pictures of whiteboards after a discussion and considering that as just enough documentation.

To make what I do useful, I make sure documentation helps me in my work (think: cost vs value) and it fills in the information need other people have. I question why they what me to document certain things to find out what they really need. Sometimes people ask for a test plan or test strategy document because they do not know what to ask and all they want to do is talk about the risks and quality of the product you are working on…

Also see question 34.

I would say that teams and organisations should make agreements with their stakeholders on how to report (which can be verbally too) and what to document. These ways of working should be evaluated on a regular basis. Situations change, teams change, ways of working change. So how we convey information naturally changes too.

In environments where teams have an agile way of working I see that less and less agreements are made on how teams should do their work and how they should document their work. I like that and I think it makes sense. Let the team decide together with their stakeholders how they want to do their job. Do not force a company standard on them without a very good reason. Tell teams what to do and why you need it, not how to do it.

I do not think guidelines for test reports nor test documentation should be documented in a test strategy (document) since the strategy is the set of ideas that guide your choice of tests. Not how you are going to report your findings. As said earlier in question 32, I don’t think we should write test plans at all unless there is a very good reason. Especially in agile environments I do not see why any body would want to write a test plan.

I think this question is answered in question 34. You als might want to read question 7.


Regression is that after a change something that used to work no longer works. It is a risk that should be dealt with. I would deal with it in the same way as I deal with all other risks: do a risk analysis and determine how to deal with that risk. So it should be a part of any test strategy since there is (almost) always a chance of regression. I would recommend to make a regression test strategy to critically think about how to deal with regression. Using the RCRCRC mnemonic by Karen Johnson might help here.

I see some teams promote all testing done for new functionality to regression tests. Especially when the checks are automated. I think that is a wrong approach. Regression tests should detect regression not retest everything you ever done before. But where is the regression coming from? “Ask how we might detect and prevent regression more quickly, less expensively, more reliably. Checks can be good, and there are other ways.” (source: Things Could Get Worse:
Ideas About Regression Testing
by Michael Bolton).

I think many teams have an obsession for regression. Therefor it is necessary to think critically of risks in your development process and deal with them appropriately.

Again: a test strategy is a process, a mental model NOT a document. The test strategy document can be as “long” as you think is useful and depends on the context. But keep in mind that often there is more than one document to visualise the test strategy since it can become pretty complex. Too complex to put in one document.

Compare this with building a house: you will create many different “strategies” to build a house: one for planning where the house will be (a map of the area), one for how the house will look when it is done (a 3D visualisation), several blueprint to build the walls, the rooms, etc., schemas for electricity, water, cables, air, list of materials, etc, etc.

A test strategy evolves and grows over time (see question 9) so if you count all documentation/visuals/models/etc. over a several sprint/iterations over months, it might become pretty big and complex. But that is not the point. The point is not how many pages, the point is how do you keep insight and overview? How do you convey the necessary information?

I use the same approach as we use in agile ways of working: splitting it up. I split up my thinking about test strategy. And I have several different visualisations to keep overview of the product and the risks involved. While testing I use a dashboard or a mind map the keep track of what I have tested, how thoroughly and what the results are (using the testing story, see question 27). I might create a separate documents if I think that suits a purpose.

Tools are of your test strategy because the strategy guides your testing, it tells you what and how to test. The how part can include thinking about tools you can use to test or make your testing easier.

I think a test strategy should be project specific. See question 6.

Should we document the strategy is already answered several times: only if it serves a purpose!Read back the earlier question since it was mentioned several times.

The second part of your question is interesting! In a way your test strategy is “implemented” in your test cases and checklists. (Please read “Test Cases Are Not Testing” by Aaron Hodder and James Bach if you tend to talk a lot about test cases). BUT test cases do not talk. To talk about our strategy with our stakeholders for purpose of determining your testing missions, planning, budget, feedback, etc. No stakeholder will read your test cases since they lack overview. Also, the test cases are the the outcome of a strategy, there is a lot of thinking done before you test.

  • What do your stakeholders need to know about the product?
  • What are the risks? How the product can fail and what are the consequences of that failure?
  • What is the product? How does it work?
  • What is the context?
  • Who is doing the testing?
  • What has been tested before?
  • What are the important aspects of the product?

Test cases do not help you structure your thoughts when thinking about what and how to test.

If I give you a pile of test cases, do you know what the test strategy is? And can you assess if the coverage is good enough for the risks identified? Can you tell a compelling story about your testing?

X coverage is how thoroughly you have examinedthe product with respect to some model of X.

Since we use risks to think about our test strategy, risk coverage is part of our thinking. Risk coverage is “how thoroughly you have examinedthe product with respect to some model of Risk
in other words: which risks have we tested? I do not see many people talk about this which is unfortunate. Test missions should be derived (amongst other things) from risks. If you start with risks, I think it is useful to report (which can be verbal, stop thinking in documents :wink:) on those risks and what you did to examine them and which risks you think are still there.

You can also think of requirements coverage: “how thoroughly you have examinedthe product with respect to some model of requirements”. In other words: which requirements have we tested? I like to talk about product coverage: “how thoroughly you have examinedthe product with respect to some model of the product”. In other words: which elements of the product did we test? To be able to talk meaningfully about coverage, we need to model the product. The Heuristic Test Strategy Model mentioned in question 1 can help you with this.

Test coverage is the sum of all coverage mentioned above.

1 Like

Nasty situation. Sad that people still ask for fluff documents. I would have talked to them about why they want the document and what they really need.

When managers start telling me what I should do and explicitly tell me how I should do that, I push back. I friendly tell them that I am very willing to help them to achieve goals, but that I think I am the expert and I will decide on how I do my work. I would also ask what problem the stuff they want in the test strategy will solve. I would talk about expectations. What do they expect me to do? If you know what they expect and why it is easier to convince them. Also I would offer them an alternative which will solve their information need.

1 Like

Sort of. Risk analysis, scope and test approach are topics discussed in the test strategy document but not necessarily chapters nor labels. We will also have to think about Oracles (means to recognise a problem) and quality criteria as well. To be able to talk meaningful about product coverage, we need a product analysis too (see question 45). I do not like the word scope, I rather talk about testing missions. The reason is that I have seen situations where testers “hide” behind scope and do not look at certain obvious risky areas nor report those risk because they are out of scope.

Look at two examples of a Test Strategy:

You will notice that both are quite different but I would still call them Test Strategies. They will guide the testing of both products. In my test strategy you will see a risk - task list. This helps me to think from the risks I see and translate them into executable tasks or test methods.

Test policy was mentioned in question 6. I am not a big fan of test policies. I think a test policy can be an inspiration document and developing one makes dialogue on how to test within a company possible. So there is some value.

But if you have responsible testers do you really need a high level document to tell them how they should do their job? If the answer is yes there is something going on in your organisation that needs fixing.

The test policies I have seen mention there should be a test strategy and dictate that a project must address which quality attributes are relevant. For me I always have a test strategy and quality attributes are always a part of my test strategy, so I do not see the value. But obviously many people do.

As often as necessary, more often than you initially think. If you have read all my previous answers in this forum, you now know that a test strategy is not a document but a mental model. A set of ideas that guide your test design or choice of tests to be performed. You also learned that a test strategy is a proces, your mental model starts small and you develop your ideas as you go. It changes constantly: you get new ideas, you change your believes based on what you learn while testing. You also know that to aid your understanding and to help you communicate the complex mental model, you can create test strategy documents.

How often do you review your thoughts?

This might feel like a philosophical answer, but you have to figure that out yourself. I would recommend to stop yourself at regular intervals and review your thoughts, look over your models, talk to other people, pair with other testers and people in your team, discuss strategies in refinements and sprint reviews, ask your clients how happy they are with the products and the team, etc.

Absolutely, absolutely, absolutely amazing work @huib.schoots. I did not expect that all questions will be answered. I kept on reading the answers and you know what each and every question was answered. I don’t have enough words to praise this work, but I felt like writing this reply. Thank you so much for those answers.

1 Like

Thank you for your kind words.

1 Like

Thank you for that detailed reply @huib.schoots. In a lot of cases in my recent experience, I have been seeing lesser and lesser focus on risks and that’s where I try to draw their attention to. Thank you once again!

1 Like