Ask Me Anything: Test Strategies

This is the same as asking “how may organisations or team within an organisation vary?”. A good Test Strategy is product specific so it definitely will vary in different teams/organisations.

Absolutely. And if the project goes as planned, please do so too! The is no way I can imagine where we create the complete Strategy upfront and we stick to it until the end. Also look at my answer for question 9. I think a Test Strategy should evolve from small and general to big and detailed over the curse of the “test project”.

For me a Test Strategy is an evolutionary (mental) model that changes all the time. When I learn new things, when I get new insights, when I see parts of it aren’t working as planned, when I get feedback that the ideas I have aren’t effective or my way of working is inefficient. I share my thoughts and (parts of) my test strategy constantly with many people I talk to or work with. They help me sharpen my ideas.

Why do you want to explain the value? Why don’t you just show the value by example? Tell them about your Test Strategy and show them artefacts to support your story.

You always have a Test Strategy in your head. Maybe you write it down, maybe you don’t. So what is the problem we are talking about here? Do you need to convince them to spend time on creating a document which captures your Strategy? Or do they refuse to help you or spend time on it?

The value of any strategy lies in knowing what to do and how to collect the information needed. The clearer your mental models become, the better you will know what to test, why and how. The strategy will create insight in which information our stakeholders want (missions), what the risks are, what the status of the product is, what actions we need to take to mitigate the risk (test or something else).

A Test Strategy is a set of ideas that guide your testing. A Test Strategy document is something different. Have a look at what Michael Bolton said about that here.

Resources can be found on my blog. Have a look at my Great Resources page. You’ll find a section on Test Strategy with many links that will interest you.

This question was answered in the AMA session. You can find the recording here. The question is answered at 51 minutes and 20 seconds.

First let’s talk about the difference between a test plan and a test strategy. To me a test strategy is a set of ideas (not a document) to guide your testing. This is not a plan. A test plan is the set of ideas that guide your test project. A Test Plan is the sum of logistics (the set of ideas that guide your application of resources to fulfilling the test strategy) and the strategy. Michael Bolton wrote a blog post about this titled “What Should A Test Plan Contain?” almost 12 years ago.

What definition says that? I googled it and found several websites that made me sad. On those pages I read that a test strategy is a high level document. They talk about a strategy being generic requirements for testing or how to test on an organisational level. I am talking about something completely different here! Please remember what Rikard Edgren said in his fabulous tutorial "Test Strategy - Next Level. It is in the combination of WHAT and HOW you find the real strategy. If you separate the WHAT and the HOW, it becomes general and quite useless.

I can not emphasize it enough, A Test Strategy is not a document! Whenever you test anything, you have a Strategy in your head. Most of your Strategy is in your head or in several heads spread over the team and will become clear when you discuss it. Writing things down, making mind maps or other visuals and models only helps you understand and communicate your Strategy better. The trick is to become aware of the Strategy in your head (mental model) and create insight and overview for yourself and your team and the stakeholders.

We have strategies for everything we do, even when we do not write them down or visualise them. Ask yourself, do I need to write it down? Why? To discuss it? To make it better? To communicate it? To report or for accountability purposes?

When you think of how to test a whole project or a separate user story, you think of different things but also about stuff that overlaps. I think it is a good idea to “split” your Strategy up and examine it from different angles: project, features, user stories, releases, etc. Using many perspectives and many models (see my answer for question 1 and question 2), will make your thinking better. Which does not mean it needs separate documents or in some cases even a document at all (a Strategy is not a document remember?).

See question 16.

Absolutely. See question 1.

Same question as question 8.

I am not sure what you are asking. What do you you mean with “a high-level test strategy for all testers?” And I am a bit confused by the second part: “specifically around the context of what you are testing?” Any test strategy is depending on context, so what kind of strategy are you thinking of not depending on any context? Maybe you are talking about a way of working?

I think of my testing strategy on different levels and from different perspectives all the time. See question 16. I can imagine that we think of testing on a high-level at the beginning of a project. Like performance is important, so we need to do performance testing at some time, so let’s make sure we have the right tooling and environment ready.

Any test strategy that stays high-level is not something I would value very much. A strategy needs give insight in WHAT you gonna test and HOW, which is pretty specific. I can imagine that when you start to work on a new project or new feature, it takes time to learn about the product. We have to deal with not knowing many details yet. This knowledge will grow and we will learn the necessary product details.

Interesting question. What do you mean by not working?

  • Not finding (enough) bugs?
  • Bugs found are not important enough?
  • Get the wrong information from testing?
  • Too expensive?
  • Not finished in time?
  • Doing too much testing?
  • Testing the wrong things
  • Using the wrong tools?
  • Not done according to company policies?
  • Environments are blocked because tests often crash the product?
  • Can’t be executed because a lack of skills?
  • Stakeholders are not happy with the test strategy?

Often only a part of the strategy is not working. Signs and symptoms depend on our missions (what do your clients, stakeholders or team expect from testing?). I think going through the list above might give you a few ideas.

Also, how do you answer the question: when do you stop testing? When is your product good enough?
James Bach wrote an blogpost “How Much is Enough? Testing as Story-Telling” and an article about “a framework for good enough testing”. Also Michael Bolton wrote an interesting blogpost about “When Do We Stop a Test?”.

I suggest to talk about testing and quality with your team and stakeholders. Do a retrospective every now and then to find out if there are improvement opportunities (which also can mean testing less). Often tell the three part testing story to your team and stakeholders and decide together if your test strategy is good enough. The three part testing story helps you gain insight in:

  1. the product story
  2. the testing story
  3. the story about the quality of testing

I cannot answer this question because I miss the context and I have no clue what kind of product we are talking about here.

Generally speaking I would learn about the product, think about were it might fail, think of ways to find those problems and think of ways to explore the product to find unknown risks. This is the same approach as I described in question 1:

  1. Missions for your testing
  2. Product analysis
  3. Oracles & information sources
  4. Quality characteristics
  5. Context: project environment
  6. Test strategies

I cannot answer this question because I miss the context and I have no clue what kind of product we are talking about here. See my answer for question 22.

See my answer at question 22.

You have to be more specific in your question. I miss the specifics of the context and I have no clue what kind of product we are talking about here.

Follow the general meta approach described earlier but adapt it for the specific context you are in (waterfall, agile or whatever) and the product you are testing.

I am afraid I do not have an example for the context you are in.

I am a bit worried by your question. If your main goal is test automation, you are doing it wrong! Automation cannot be the mission in any test strategy. I am not saying you shouldn’t do automation. I think any project could benefit from automation. But it needs to solve a problem. Automation in testing is never a solution in itself.

I would suggest a couple of heuristics for sustainable automation:

  • make automation part of your daily work and a team responsibility
  • tread the test code as production code
  • have people working on it who have to right skills (automation = programming, what to automate = testing skill)
  • think what feedback loops you need (what information you need to collect)

Also have a look at this:

When I summarise my testing I use the three part testing story. The testing story helps me summarise my knowledge about:

  1. the product
  2. the testing I have done
  3. the quality of my testing

The amount of detail depends on who I am talking to: developers, testers, managers, directors, clients and what they want to know. So I often start my reporting with asking what they would like to know.

Test Strategy is a solution to a complex problem: how do we meet the information needs of our team & stakeholders in the most efficient way possible? A Test Strategy is the start for every tester and helps you to do the right things, gives testing structure and provides insight in risks and coverage.

A Test Strategy is not a document but “a set of ideas that guide the choice of tests to be performed”. (Rapid Software Testing).

Test strategy contains the ideas that guide your testing effort and deals with what to test, and how to do it. It is in the combination of WHAT and HOW you find the real strategy. If you separate the WHAT and the HOW, it becomes general and quite useless. (Rikard Edgren).

I would learn about the product, think about were it might fail, think of ways to find those problems and think of ways to explore the product to find unknown risks.

If you need guidance on how to create a Test Strategy do these activities:

  1. Missions for your testing
  2. Product analysis
  3. Oracles & information sources
  4. Quality characteristics
  5. Context: project environment
  6. Test strategies

Be aware that the activities do not need to be done specifically in this order. Most likely you will do this in an iterative way, building your evolutionary Test Strategy as you go.

Welcome! Thank you for watching and asking your question!

There are many stakeholders for testing and therefore your test strategy, among which are:

  • Your team
  • Other teams
  • Managers (and managers of those managers)
  • Clients
  • Users
  • Maintenance/OPS
  • Business departments
  • Auditors

Have a look at question 7 on how to get them involved.