Ask Me Anything: Test Strategies

This is a bit off topic, but I am happy to answer your question. That is, if I understand what you actually asking.

  1. Do you want to know how to become better in automation in testing?
  2. Do you want to know how to make automation in testing part of your test strategy?
  3. Or do you want to know how to improve the performance of your automated checks?

Ad 1) Have a look at these AMA by Richard Bradshaw or Mark Winteringham. Maybe read this post by Bas Dijkstra.

Ad 2) By collaborating with the whole team. The most important thing you can do is talk about the risks in the whole development lifecycle with your team. And discuss measures to deal with them. Start with why you want to automate. Talk about that first. Automation is not a solution on it self, it must solve a specific problem.

Talk to your developers about what you want to test and ask how they would do it. Ask for help form people who have experience in setting up automation to assist their testing in a way that solves your problem. Experiment and find a way to find out if your solution is successful. Talk about testability. Also have a look at question 26.

Ad 3) This is not a question I can answer without getting some context form you.

First by NOT thinking of a Test Strategy as an artefact. That is very helpful in understanding that there is a lot more going on than we write down. Tacit knowledge and the use of heuristics is very important in dealing with complexity.

Secondly, I do not think there is a true look of any document. That implies there is one way to deal with documentation. I think how to document and what to document highly depends on the situation.

Looking at documenting my test strategy, I like to keep them lean by adding visualisations. I keep them lean by not writing down everything and telling good stories about my work to the people I work with. I keep my documentation lean by using notepad more that MS Word so that I do not focus on format and lay-out too much. I keep documentation lean by taking pictures of whiteboards after a discussion and considering that as just enough documentation.

To make what I do useful, I make sure documentation helps me in my work (think: cost vs value) and it fills in the information need other people have. I question why they what me to document certain things to find out what they really need. Sometimes people ask for a test plan or test strategy document because they do not know what to ask and all they want to do is talk about the risks and quality of the product you are working on…

Also see question 34.

I would say that teams and organisations should make agreements with their stakeholders on how to report (which can be verbally too) and what to document. These ways of working should be evaluated on a regular basis. Situations change, teams change, ways of working change. So how we convey information naturally changes too.

In environments where teams have an agile way of working I see that less and less agreements are made on how teams should do their work and how they should document their work. I like that and I think it makes sense. Let the team decide together with their stakeholders how they want to do their job. Do not force a company standard on them without a very good reason. Tell teams what to do and why you need it, not how to do it.

I do not think guidelines for test reports nor test documentation should be documented in a test strategy (document) since the strategy is the set of ideas that guide your choice of tests. Not how you are going to report your findings. As said earlier in question 32, I don’t think we should write test plans at all unless there is a very good reason. Especially in agile environments I do not see why any body would want to write a test plan.

I think this question is answered in question 34. You als might want to read question 7.

.

Regression is that after a change something that used to work no longer works. It is a risk that should be dealt with. I would deal with it in the same way as I deal with all other risks: do a risk analysis and determine how to deal with that risk. So it should be a part of any test strategy since there is (almost) always a chance of regression. I would recommend to make a regression test strategy to critically think about how to deal with regression. Using the RCRCRC mnemonic by Karen Johnson might help here.

I see some teams promote all testing done for new functionality to regression tests. Especially when the checks are automated. I think that is a wrong approach. Regression tests should detect regression not retest everything you ever done before. But where is the regression coming from? “Ask how we might detect and prevent regression more quickly, less expensively, more reliably. Checks can be good, and there are other ways.” (source: Things Could Get Worse:
Ideas About Regression Testing
by Michael Bolton).

I think many teams have an obsession for regression. Therefor it is necessary to think critically of risks in your development process and deal with them appropriately.

Again: a test strategy is a process, a mental model NOT a document. The test strategy document can be as “long” as you think is useful and depends on the context. But keep in mind that often there is more than one document to visualise the test strategy since it can become pretty complex. Too complex to put in one document.

Compare this with building a house: you will create many different “strategies” to build a house: one for planning where the house will be (a map of the area), one for how the house will look when it is done (a 3D visualisation), several blueprint to build the walls, the rooms, etc., schemas for electricity, water, cables, air, list of materials, etc, etc.

A test strategy evolves and grows over time (see question 9) so if you count all documentation/visuals/models/etc. over a several sprint/iterations over months, it might become pretty big and complex. But that is not the point. The point is not how many pages, the point is how do you keep insight and overview? How do you convey the necessary information?

I use the same approach as we use in agile ways of working: splitting it up. I split up my thinking about test strategy. And I have several different visualisations to keep overview of the product and the risks involved. While testing I use a dashboard or a mind map the keep track of what I have tested, how thoroughly and what the results are (using the testing story, see question 27). I might create a separate documents if I think that suits a purpose.

Tools are of your test strategy because the strategy guides your testing, it tells you what and how to test. The how part can include thinking about tools you can use to test or make your testing easier.

I think a test strategy should be project specific. See question 6.

Should we document the strategy is already answered several times: only if it serves a purpose!Read back the earlier question since it was mentioned several times.

The second part of your question is interesting! In a way your test strategy is “implemented” in your test cases and checklists. (Please read “Test Cases Are Not Testing” by Aaron Hodder and James Bach if you tend to talk a lot about test cases). BUT test cases do not talk. To talk about our strategy with our stakeholders for purpose of determining your testing missions, planning, budget, feedback, etc. No stakeholder will read your test cases since they lack overview. Also, the test cases are the the outcome of a strategy, there is a lot of thinking done before you test.

  • What do your stakeholders need to know about the product?
  • What are the risks? How the product can fail and what are the consequences of that failure?
  • What is the product? How does it work?
  • What is the context?
  • Who is doing the testing?
  • What has been tested before?
  • What are the important aspects of the product?

Test cases do not help you structure your thoughts when thinking about what and how to test.

If I give you a pile of test cases, do you know what the test strategy is? And can you assess if the coverage is good enough for the risks identified? Can you tell a compelling story about your testing?

X coverage is how thoroughly you have examinedthe product with respect to some model of X.

Since we use risks to think about our test strategy, risk coverage is part of our thinking. Risk coverage is “how thoroughly you have examinedthe product with respect to some model of Risk
in other words: which risks have we tested? I do not see many people talk about this which is unfortunate. Test missions should be derived (amongst other things) from risks. If you start with risks, I think it is useful to report (which can be verbal, stop thinking in documents :wink:) on those risks and what you did to examine them and which risks you think are still there.

You can also think of requirements coverage: “how thoroughly you have examinedthe product with respect to some model of requirements”. In other words: which requirements have we tested? I like to talk about product coverage: “how thoroughly you have examinedthe product with respect to some model of the product”. In other words: which elements of the product did we test? To be able to talk meaningfully about coverage, we need to model the product. The Heuristic Test Strategy Model mentioned in question 1 can help you with this.

Test coverage is the sum of all coverage mentioned above.

1 Like

Nasty situation. Sad that people still ask for fluff documents. I would have talked to them about why they want the document and what they really need.

When managers start telling me what I should do and explicitly tell me how I should do that, I push back. I friendly tell them that I am very willing to help them to achieve goals, but that I think I am the expert and I will decide on how I do my work. I would also ask what problem the stuff they want in the test strategy will solve. I would talk about expectations. What do they expect me to do? If you know what they expect and why it is easier to convince them. Also I would offer them an alternative which will solve their information need.

1 Like

Sort of. Risk analysis, scope and test approach are topics discussed in the test strategy document but not necessarily chapters nor labels. We will also have to think about Oracles (means to recognise a problem) and quality criteria as well. To be able to talk meaningful about product coverage, we need a product analysis too (see question 45). I do not like the word scope, I rather talk about testing missions. The reason is that I have seen situations where testers “hide” behind scope and do not look at certain obvious risky areas nor report those risk because they are out of scope.

Look at two examples of a Test Strategy:

You will notice that both are quite different but I would still call them Test Strategies. They will guide the testing of both products. In my test strategy you will see a risk - task list. This helps me to think from the risks I see and translate them into executable tasks or test methods.

Test policy was mentioned in question 6. I am not a big fan of test policies. I think a test policy can be an inspiration document and developing one makes dialogue on how to test within a company possible. So there is some value.

But if you have responsible testers do you really need a high level document to tell them how they should do their job? If the answer is yes there is something going on in your organisation that needs fixing.

The test policies I have seen mention there should be a test strategy and dictate that a project must address which quality attributes are relevant. For me I always have a test strategy and quality attributes are always a part of my test strategy, so I do not see the value. But obviously many people do.

As often as necessary, more often than you initially think. If you have read all my previous answers in this forum, you now know that a test strategy is not a document but a mental model. A set of ideas that guide your test design or choice of tests to be performed. You also learned that a test strategy is a proces, your mental model starts small and you develop your ideas as you go. It changes constantly: you get new ideas, you change your believes based on what you learn while testing. You also know that to aid your understanding and to help you communicate the complex mental model, you can create test strategy documents.

How often do you review your thoughts?

This might feel like a philosophical answer, but you have to figure that out yourself. I would recommend to stop yourself at regular intervals and review your thoughts, look over your models, talk to other people, pair with other testers and people in your team, discuss strategies in refinements and sprint reviews, ask your clients how happy they are with the products and the team, etc.

Absolutely, absolutely, absolutely amazing work @huib.schoots. I did not expect that all questions will be answered. I kept on reading the answers and you know what each and every question was answered. I don’t have enough words to praise this work, but I felt like writing this reply. Thank you so much for those answers.

1 Like

Thank you for your kind words.

1 Like

Thank you for that detailed reply @huib.schoots. In a lot of cases in my recent experience, I have been seeing lesser and lesser focus on risks and that’s where I try to draw their attention to. Thank you once again!

1 Like