Tonight, @huib.schoots joined us for an Ask Me Anything all about Test Strategies.
Of course, any resources or questions we didnât get to on the night will be added here but if Iâve missed any or you have another question youâd like to ask, add it to the thread 
If you missed the live session, a recording will be available on the Ministry of Testing website once weâve edited it and added captions.
Test Strategy discussions on The Club
Huibs blog http://www.huibschoots.nl/
How elaborative Test Strategies need to be is very context specific. Also: a Test Strategy is not a document but a mental model, an ongoing proces to guide your testing as I explained in the AMA session. Documents and visuals help you understand and transfer your ideas. A Test Strategy needs give insight in WHAT you gonna test and HOW. I made an example in my blogpost âCreating a Test Strategyâ. I would not recommend using a template, because every situation is different. Also I am a bit reluctant to use templates, since they often become an exercise in filling in the blanks, instead of through (critical and creative) thinking about what we need.
If you need guidance on how to create a Test Strategy do the activities (read: build the models) described in my blog post. Be aware that the activities do not need to be done specifically in this order. Most likely you will do this in an iterative way, building your evolutionary Test Strategy as you go.
All these things can be found in the Heuristic Test Strategy Model (HTSM) by James Bach.
Also have a look at the interesting and well worded reply by Michael Bolton on this forum.
In the past I made my strategies visual in many different ways. Be creative and be concise. Do not write (heavy) documents if you do not have to. Only create artefacts that serve a purpose and add value to what you do. Most of the time people will not read them anyway. Some ideas to visualise your thinking: have a look at my collection of Great Resources on visualisation.
See my answer for question 1. Start with the HTSM. For understanding the product, the progress, the coverage and what is going on in discussions in your team, there are many models you can use. I recommend to use many models, since every model gives you a different perspective. There is âunfortunately not 1 model to rule them allâ. Some examples:
More on models in these blogpost:
Interesting question which I cannot answer, since I do not know enough about API performance testing. I hope there are other people here on the Club who can answer this question for you.
I think there is always time for strategising, and I even dare to claim that you always take time to strategise. But you might not write a strategy document. You do the strategic thinking, bu you are probably struggling how to visualise it or write it down.
Keeping pace with each other is what I call collaboration. Keeping pace with development is a team responsibility. Which means that if you cannot keep up with the rest of the team, they are going too fast and they are creating too many risks that need to be taken care of by testing. Build a quality culture. To solve this issue you need to talk about your way of working with the team. Some examples of what you could consider:
To be able to test deeply, you need to know WHAT you want to test deeply. To figure out what is important, you need a Test Strategy. How do you know what to test? What is important? What is risky? Think about the risks in the product.
I think that creating a Test Strategy is essential to do excellent testing. Creating a test strategy document is not. Use concise and lean ways of capturing and transferring your Test Strategy: use mind maps, drawings on a white board, flip charts and sticky notes. Stop putting Test Strategies in Word documents. If you want to create a list, consider using notepad!
I am not sure if I understand your question.
Part of deciding HOW to test is related to the skills of the people executing the activities in the Test Strategy. But in essence it makes no difference. Still you have to figure out what the product needs to do, where the risks are and how to mitigate them. If your case the WHO does the testing is different.
If the developers have enough testing skills (and motivation) to do the testing, I see no problem. In Rapid Software Testing we call people who can do skilled testing responsible testers. In some cases developers do not like testing and they lack motivation to learn or to do serious testing. So a âresponsible testerâ should supervise the testing. This means that the tester knows what coverage is needed and helps his/her team members to do the testing that is needed.
That is something I would probably call a âTest Policyâ. A Test Policy in my experience is a document in which, often big companies, try to explain what (the purpose of) testing is and what in their way of working should be adhered by teams. In my experience companies write it but teams never really look at it. It is often on such a general level that even in audits no deficiencies are found. I ask myself why do we need those documents if nobody really reads them or uses them? Also a thought that pops up in my mind is: do developers or managers have document in which they explain what the purpose of development or management is?
A Test Strategy is a solution to a complex problem: how do we meet the information needs of our team/project & stakeholders in the most efficient way possible? A Test Strategy is âa set of ideas that guide your test design or choice of tests to be performedâ.
Rikard Edgren says: âTest strategy contains the ideas that guide your testing effort and deals with what to test, and how to do it. (Some people say test plan or test process, which is unfortunateâŚ). It is in the combination of WHAT and HOW you find the real strategy. If you separate the WHAT and the HOW, it becomes general and quite useless.â
I cannot imagine that you want to define a Test Strategy on company level, unless you are talking about testing the whole landscape of applications and their interaction and flow in chain testing? In that case my Test Strategy would probably have the items I mentioned in question 1.
I think this topic was covered in the AMA. But let me still answer this, since this is an important topic and I like to add a few things. Involving people is often a matter of getting their interest. So talking in tester slang is not going to help you.
So do not say stuff like:
Weâve will create 17 test cases in the system test, we will automate 50% of those test cases for area C so we will have 30% code coverage. Last time in this area we found three major and five medium bugs, so I want to add 3 FTEâs to do orthogonal pairwise testing. To mitigate the risk of this calculation to fail, we should do equivalence class analysis, use self-verifying data and do the elementary comparison testing!
Also talking about stuff that is too general is boring because it will not add a lot of value. Talking in too much detail is also boring for many people because they want to talk about stuff that matters to them. So I suggest to talk to different stakeholder in different settings.
Alex Schladebeck and I did a keynote about how to talk about testing and quality called âLetâs stop talking about testing, letâs start thinking about valueâ. Check it out, although it is not recorded, we wrote a blogpost about it with the main points.
First premise: letâs distinguish between a strategy and a strategy document. The specific nature of that difference seems to have been missing from all of this discussion so far. A test strategy (for anything, including for an API) is the set of ideas that guide your choice of tests. (A test strategy document is some document that represents these ideas to some degree. A strategy is not a document.)
Second premise: you have choices in testing; everything that you do in testing represents a choice to do some things, and choice not to do other things. By âchoice of testsâ here, we donât mean âselecting test casesâ; we mean the choices you make about risks you want to investigate, quality criteria you want to consider, techniques you can apply, aspects of the product that you want to cover, data that you might selectâŚ
Third premise: to guide is to influence, but not to determine or to dictate. Elements of your context will enable or constrain your choices; so will your skills and your mindset.
Fourth premise: as a heuristic, for any given X, âX testingâ is âtesting focused on risk related to Xâ.
Now: if you are even considering âAPI performance testingâ, you already have a strategy; that is, you already have a set of ideas that might guide your choice of tests. That set of ideas may be vague or confused, or it may be sharp and rich and coherent, but youâve got at least two notions in your head: âAPIâ and âperformanceâ.
The API is an interface. Itâs a means of getting at functions in a product. So in one sense, youâre not testing the API; youâre testing the product, using the API. To test the product, you need models of the product. On the other hand, you can also think of the API as a product in itself (someone produced it). To test that product, you need models of that too.
âPerformanceâ is a word that represents a category of ideas that people might value in a product; itâs a quality criterion. Performance might include notions of speed; responsiveness; capacity; reliability and robustness under load, or under stress, or under some kind of constraint.
The idea of interface as an element of the product and performance as a quality criterion can be found in the Heuristic Test Strategy Model (https://www.satisfice.com/download/heuristic-test-strategy-model), which provides us with a set of ideas about how to think of strategy generally.
That leads us to key questions that guide testing related to performance, via the API:
Do we know what the customer wants? How big a deal is performance as part of that? (Maybe itâs not a big deal, and we donât have to do a ton of performance testing.) Do customers use the API? Which customers? Or is the API simply a means that developers use (and that I can use) for easy access to aspects of the product for which performance is critical? Have we included such things in the design of the product and of its API?
As the developers are building the product, are they building it with performance in mind? As theyâre building it, is the performance of the product they think it is? What testing are the programmers doing? Are they reviewing the design and the code for performance-related issues? Do the low-level checks that theyâre doing include some focus on timing, or on stress? Are there specific functions, accessible through the API, on which they are doing performance analysis? Are they well-documented?
What do we need to do to prepare to test for performance-related problems? What tools do we have? What tools might we need to develop? Is there logging built into the product? Do we need write code and use parts of the API for deeper analysis? Have we prepared representative environments and representative data to model real-world performance?
How might we explore or experiment with the product to examine performance-related risk? How might we stress the product to extremes? How might we starve the product of things that it needs?
It seems to me that examining and exploring these questions will help to guide you to develop a set of ideas for testing performance.
This might help too: https://www.developsense.com/blog/2018/07/exploratory-testing-on-an-api-part-1/
I think Michael an important point here - these are the questions that werenât addressed in the AMA last night. Some of the points youâve mentioned were addressed and clarified last night. The video will be live later this week for watching.
Metrics are not the first thing I think about when thinking of a Test Strategy. A Test Strategy helps me guide my testing. Drive the choice of testing I do. How do metrics play a part in that?
I have a love/hate relation with metrics. This is because I think it is generally a good idea to have metrics to measure useful and meaningful thing that drive value. That help me make decisions about the way we work or the product we are working on. On the other hand, we should be really careful with metrics. There are many really bad metrics.
What problem are you trying to solve or what insight do you want to gain? Which data could help you make decisions? Try to find metrics that support that. Have a look at the links I provided on âmeaningful and bad metricsâ to get an idea what could help you.
I cannot imagine that anybody would have a fixed strategy, since we learn new things all the time. So I would expect the Strategy to change too. So yes, it develops. A Strategy starts small. From the start to the finish of a test project or testing a release or even a user story, we start small. How to develop (and decide about) a Strategy is described in question 1.
So a Test Strategy evolves, it grows while we learn & discover more about the product. Based on risks we decide what to test first. For those areas we have more details in the Test Strategy. We go from global to more details. Your Test Strategy is âcompleteâ at the end of your testing.
This is the same as asking âhow may organisations or team within an organisation vary?â. A good Test Strategy is product specific so it definitely will vary in different teams/organisations.
Absolutely. And if the project goes as planned, please do so too! The is no way I can imagine where we create the complete Strategy upfront and we stick to it until the end. Also look at my answer for question 9. I think a Test Strategy should evolve from small and general to big and detailed over the curse of the âtest projectâ.
For me a Test Strategy is an evolutionary (mental) model that changes all the time. When I learn new things, when I get new insights, when I see parts of it arenât working as planned, when I get feedback that the ideas I have arenât effective or my way of working is inefficient. I share my thoughts and (parts of) my test strategy constantly with many people I talk to or work with. They help me sharpen my ideas.
Why do you want to explain the value? Why donât you just show the value by example? Tell them about your Test Strategy and show them artefacts to support your story.
You always have a Test Strategy in your head. Maybe you write it down, maybe you donât. So what is the problem we are talking about here? Do you need to convince them to spend time on creating a document which captures your Strategy? Or do they refuse to help you or spend time on it?
The value of any strategy lies in knowing what to do and how to collect the information needed. The clearer your mental models become, the better you will know what to test, why and how. The strategy will create insight in which information our stakeholders want (missions), what the risks are, what the status of the product is, what actions we need to take to mitigate the risk (test or something else).
A Test Strategy is a set of ideas that guide your testing. A Test Strategy document is something different. Have a look at what Michael Bolton said about that here.
Resources can be found on my blog. Have a look at my Great Resources page. Youâll find a section on Test Strategy with many links that will interest you.
This question was answered in the AMA session. You can find the recording here. The question is answered at 51 minutes and 20 seconds.
First letâs talk about the difference between a test plan and a test strategy. To me a test strategy is a set of ideas (not a document) to guide your testing. This is not a plan. A test plan is the set of ideas that guide your test project. A Test Plan is the sum of logistics (the set of ideas that guide your application of resources to fulfilling the test strategy) and the strategy. Michael Bolton wrote a blog post about this titled âWhat Should A Test Plan Contain?â almost 12 years ago.
What definition says that? I googled it and found several websites that made me sad. On those pages I read that a test strategy is a high level document. They talk about a strategy being generic requirements for testing or how to test on an organisational level. I am talking about something completely different here! Please remember what Rikard Edgren said in his fabulous tutorial "Test Strategy - Next Level. It is in the combination of WHAT and HOW you find the real strategy. If you separate the WHAT and the HOW, it becomes general and quite useless.
I can not emphasize it enough, A Test Strategy is not a document! Whenever you test anything, you have a Strategy in your head. Most of your Strategy is in your head or in several heads spread over the team and will become clear when you discuss it. Writing things down, making mind maps or other visuals and models only helps you understand and communicate your Strategy better. The trick is to become aware of the Strategy in your head (mental model) and create insight and overview for yourself and your team and the stakeholders.
We have strategies for everything we do, even when we do not write them down or visualise them. Ask yourself, do I need to write it down? Why? To discuss it? To make it better? To communicate it? To report or for accountability purposes?
When you think of how to test a whole project or a separate user story, you think of different things but also about stuff that overlaps. I think it is a good idea to âsplitâ your Strategy up and examine it from different angles: project, features, user stories, releases, etc. Using many perspectives and many models (see my answer for question 1 and question 2), will make your thinking better. Which does not mean it needs separate documents or in some cases even a document at all (a Strategy is not a document remember?).
