Ask Me Anything: Test Strategies

Tonight, @huib.schoots joined us for an Ask Me Anything all about Test Strategies.

Of course, any resources or questions we didn’t get to on the night will be added here but if I’ve missed any or you have another question you’d like to ask, add it to the thread :slight_smile:

If you missed the live session, a recording will be available on the Ministry of Testing website once we’ve edited it and added captions.

3 Likes

Resources that came up in the talk or the chat



Test Strategy discussions on The Club

Huibs blog http://www.huibschoots.nl/

Questions we didn’t get to live

  1. How elaborate a test Strategy should be? Could u show us a template for test strategy to be used in Agile.
  2. I like using models when developing strategy, have you any models you would suggest?
  3. What is the test strategies can I apply in API performance testing?
  4. With agile and keeping pace with development, how do you solve the problem of testing deeply but quickly, where there isn’t even time to spend on strategising?
  5. Do you have any advice on best test strategy when Quality Assistance approach is being followed e.g. Developers doing the majority of the testing with guidance and advice given by the QAs.
  6. What content would you expect to see in an Over Arching Company Test Strategy?
  7. How do I get my stakeholders involved?
  8. what sort of metrics do you find valuable in a test strategy?
  9. Should there be a fixed strategy? Or is it something that should develop over time? How should the strategy be decided?
  10. How might a strategy vary between different organisations or teams within an organisation?
  11. When a project does not go as planned and the documented test strategy can no longer be followed should you go back and revise the strategy?
  12. How do you explain the value gained by starting with a test strategy to a PO / Scrum team / other testers?
  13. Do you have any resources to suggest for those who are new to writing a test strategy, (other than this wonderful session)?
  14. Do we need separate test Strategy document for manual and automation testing
  15. The definition says that Test Plan is derived from Software Requirements Specification (SRS). How would you suggest to create the Plan in case of the absence of the SRS?
  16. Test strategies when do you write one? Is it at project or feature level does it include all the work in a sprint deliverable?
  17. Do you have an interesting ways of presenting test strategies rather than the standard word document?
  18. Can you distinguish between a “Test Strategy” and a “Test Strategy Document”?
  19. what sort of metrics do you find valuable in a test strategy?
  20. Would you suggest having a high-level test strategy for all testers on the team AND more granular test strategies specifically around the context of what you are testing?
  21. How can you tell when the current test strategy is no longer working? What are the signs and symptoms?
  22. How should a test strategy be approached when testing embedded software systems on targeted hardware (i.e. a measurement solution where both HW and SW has to be tested)?
  23. Which test strategy can be approached with APIs and AMQ Microservices and check the register in DB?
  24. How to integrate Hardware testing (electronics, mechanics…) in a Testing Strategy?
  25. How do you deal in a Test Strategy with different development models (eg.: waterfall or repeated mini waterfall model for hardware component and agile for embedded or application Software development? Do you have any examples for this context?
  26. Please, could you provide your guidelines about what to consider if main goal is implement a solid testing automation? (solid meaning it can grow sustainably)
  27. Bare basics - what is the absolute minimum amount of information required for a test summary to be effective - MVP of test summaries
  28. How would you convey the essence of testing strategies to someone who’s completely new to testing?
  29. Hey Huib…Thank you for this session. Who are the various stakeholders of a Test Strategy, and how can we get their engagement and buy in?
  30. Should a Test Strategy for an organisation be methodology agnostic?
  31. What would you do, if you put in place a test strategy/guide for a project then realise it’s not working or its not effective?
  32. So is test strategy defined in a test plan document since former is not documented.
  33. Can you provide an example of a bad test strategy? What went wrong and what did you learn from it?
  34. what techniques/ actions can be employed to ensure a Test Strategy is being adhered to all levels?
  35. what is the difference between test plan and test strategy…also, in organizations, do both test plan and test strategy documents co-exist?
  36. for someone who has done a manual testing in his/her entire career, what do you recommend for him/her to do first to speed up on test automation?
  37. In today’s software quality space what is a true look of test strategy/plan artifact and how do you make it lean & useful.
  38. Do guidelines for test reports / test documentation belong into a test strategy or in a test plan?
  39. How do you get non Testers to buy into the organisational Test Strategy?
  40. How does one account for regression in a test strategy?
  41. as u said that test strategy is only what to test and how to test, how much long u anticipate the document is…should be 2-3 pager I guess…please clarify
  42. Where do test tools fit into the test strategy? I am thinking about tools used for manual and automated tests.
  43. I’ve seen some companies that have a company wide test strategy that applies to all their projects or should this be project/program based?
  44. Should we document the strategy? OR it will be implicitly implemented in our test cases or test checklist as the strategy is context?
  45. What do you think should be the primary focus area of a test strategy…Test coverage or Risk coverage?
  46. While working in an agile project, I was asked to create a test strategy. Our main stakeholder (and product owner) requested me to put stuff into it that belonged into a test plan. Since they insisted and I could not convince them otherwise, we ended up with a long “fluffy document” as you have called it earlier. How would you have handled this situation?
  47. so the labels in test strategy documents are: scope, test approach, risk analysis…right?
  48. Do you think a test policy is important for a company? How is this related to a Test strategy?
  49. How often would you recommend to review the test strategy?
2 Likes

How elaborative Test Strategies need to be is very context specific. Also: a Test Strategy is not a document but a mental model, an ongoing proces to guide your testing as I explained in the AMA session. Documents and visuals help you understand and transfer your ideas. A Test Strategy needs give insight in WHAT you gonna test and HOW. I made an example in my blogpost “Creating a Test Strategy”. I would not recommend using a template, because every situation is different. Also I am a bit reluctant to use templates, since they often become an exercise in filling in the blanks, instead of through (critical and creative) thinking about what we need.

If you need guidance on how to create a Test Strategy do the activities (read: build the models) described in my blog post. Be aware that the activities do not need to be done specifically in this order. Most likely you will do this in an iterative way, building your evolutionary Test Strategy as you go.

  1. Missions for your testing
  2. Product analysis
  3. Oracles & information sources
  4. Quality characteristics
  5. Context: project environment
  6. Test strategies

All these things can be found in the Heuristic Test Strategy Model (HTSM) by James Bach.

Also have a look at the interesting and well worded reply by Michael Bolton on this forum.

In the past I made my strategies visual in many different ways. Be creative and be concise. Do not write (heavy) documents if you do not have to. Only create artefacts that serve a purpose and add value to what you do. Most of the time people will not read them anyway. Some ideas to visualise your thinking: have a look at my collection of Great Resources on visualisation.

See my answer for question 1. Start with the HTSM. For understanding the product, the progress, the coverage and what is going on in discussions in your team, there are many models you can use. I recommend to use many models, since every model gives you a different perspective. There is “unfortunately not 1 model to rule them all”. Some examples:

  • Requirements
  • Technical designs
  • Stakeholder overview
  • Quality attributes
  • Architectural overviews
  • Proces models
  • Flow charts
  • Data models
  • Risk models
  • Timelines
  • Your own product coverage outlines (see my video on product coverage outlines on youtube)

More on models in these blogpost:

Interesting question which I cannot answer, since I do not know enough about API performance testing. I hope there are other people here on the Club who can answer this question for you.

I think there is always time for strategising, and I even dare to claim that you always take time to strategise. But you might not write a strategy document. You do the strategic thinking, bu you are probably struggling how to visualise it or write it down.

Keeping pace with each other is what I call collaboration. Keeping pace with development is a team responsibility. Which means that if you cannot keep up with the rest of the team, they are going too fast and they are creating too many risks that need to be taken care of by testing. Build a quality culture. To solve this issue you need to talk about your way of working with the team. Some examples of what you could consider:

  • Talk about testability (see resources in the nice list “resources that came up in the talk or the chat”) with your team.
  • Do risk analysis during refinements to engage the whole team.
  • Add Test Strategy to your definition of Ready so you “force” your team to think about testing before the software gets build.

To be able to test deeply, you need to know WHAT you want to test deeply. To figure out what is important, you need a Test Strategy. How do you know what to test? What is important? What is risky? Think about the risks in the product.

I think that creating a Test Strategy is essential to do excellent testing. Creating a test strategy document is not. Use concise and lean ways of capturing and transferring your Test Strategy: use mind maps, drawings on a white board, flip charts and sticky notes. Stop putting Test Strategies in Word documents. If you want to create a list, consider using notepad!

I am not sure if I understand your question.

Part of deciding HOW to test is related to the skills of the people executing the activities in the Test Strategy. But in essence it makes no difference. Still you have to figure out what the product needs to do, where the risks are and how to mitigate them. If your case the WHO does the testing is different.

If the developers have enough testing skills (and motivation) to do the testing, I see no problem. In Rapid Software Testing we call people who can do skilled testing responsible testers. In some cases developers do not like testing and they lack motivation to learn or to do serious testing. So a “responsible tester” should supervise the testing. This means that the tester knows what coverage is needed and helps his/her team members to do the testing that is needed.

That is something I would probably call a “Test Policy”. A Test Policy in my experience is a document in which, often big companies, try to explain what (the purpose of) testing is and what in their way of working should be adhered by teams. In my experience companies write it but teams never really look at it. It is often on such a general level that even in audits no deficiencies are found. I ask myself why do we need those documents if nobody really reads them or uses them? Also a thought that pops up in my mind is: do developers or managers have document in which they explain what the purpose of development or management is?

A Test Strategy is a solution to a complex problem: how do we meet the information needs of our team/project & stakeholders in the most efficient way possible? A Test Strategy is “a set of ideas that guide your test design or choice of tests to be performed”.

Rikard Edgren says: “Test strategy contains the ideas that guide your testing effort and deals with what to test, and how to do it. (Some people say test plan or test process, which is unfortunate…). It is in the combination of WHAT and HOW you find the real strategy. If you separate the WHAT and the HOW, it becomes general and quite useless.”

I cannot imagine that you want to define a Test Strategy on company level, unless you are talking about testing the whole landscape of applications and their interaction and flow in chain testing? In that case my Test Strategy would probably have the items I mentioned in question 1.

I think this topic was covered in the AMA. But let me still answer this, since this is an important topic and I like to add a few things. Involving people is often a matter of getting their interest. So talking in tester slang is not going to help you.

So do not say stuff like:

We’ve will create 17 test cases in the system test, we will automate 50% of those test cases for area C so we will have 30% code coverage. Last time in this area we found three major and five medium bugs, so I want to add 3 FTE’s to do orthogonal pairwise testing. To mitigate the risk of this calculation to fail, we should do equivalence class analysis, use self-verifying data and do the elementary comparison testing!

Also talking about stuff that is too general is boring because it will not add a lot of value. Talking in too much detail is also boring for many people because they want to talk about stuff that matters to them. So I suggest to talk to different stakeholder in different settings.

Alex Schladebeck and I did a keynote about how to talk about testing and quality called “Let’s stop talking about testing, let’s start thinking about value”. Check it out, although it is not recorded, we wrote a blogpost about it with the main points.

1 Like

First premise: let’s distinguish between a strategy and a strategy document. The specific nature of that difference seems to have been missing from all of this discussion so far. A test strategy (for anything, including for an API) is the set of ideas that guide your choice of tests. (A test strategy document is some document that represents these ideas to some degree. A strategy is not a document.)

Second premise: you have choices in testing; everything that you do in testing represents a choice to do some things, and choice not to do other things. By “choice of tests” here, we don’t mean “selecting test cases”; we mean the choices you make about risks you want to investigate, quality criteria you want to consider, techniques you can apply, aspects of the product that you want to cover, data that you might select…

Third premise: to guide is to influence, but not to determine or to dictate. Elements of your context will enable or constrain your choices; so will your skills and your mindset.

Fourth premise: as a heuristic, for any given X, “X testing” is “testing focused on risk related to X”.

Now: if you are even considering “API performance testing”, you already have a strategy; that is, you already have a set of ideas that might guide your choice of tests. That set of ideas may be vague or confused, or it may be sharp and rich and coherent, but you’ve got at least two notions in your head: “API” and “performance”.

The API is an interface. It’s a means of getting at functions in a product. So in one sense, you’re not testing the API; you’re testing the product, using the API. To test the product, you need models of the product. On the other hand, you can also think of the API as a product in itself (someone produced it). To test that product, you need models of that too.

“Performance” is a word that represents a category of ideas that people might value in a product; it’s a quality criterion. Performance might include notions of speed; responsiveness; capacity; reliability and robustness under load, or under stress, or under some kind of constraint.

The idea of interface as an element of the product and performance as a quality criterion can be found in the Heuristic Test Strategy Model (https://www.satisfice.com/download/heuristic-test-strategy-model), which provides us with a set of ideas about how to think of strategy generally.

That leads us to key questions that guide testing related to performance, via the API:

  • Do we know what the customer wants? How big a deal is performance as part of that? (Maybe it’s not a big deal, and we don’t have to do a ton of performance testing.) Do customers use the API? Which customers? Or is the API simply a means that developers use (and that I can use) for easy access to aspects of the product for which performance is critical? Have we included such things in the design of the product and of its API?

  • As the developers are building the product, are they building it with performance in mind? As they’re building it, is the performance of the product they think it is? What testing are the programmers doing? Are they reviewing the design and the code for performance-related issues? Do the low-level checks that they’re doing include some focus on timing, or on stress? Are there specific functions, accessible through the API, on which they are doing performance analysis? Are they well-documented?

  • What do we need to do to prepare to test for performance-related problems? What tools do we have? What tools might we need to develop? Is there logging built into the product? Do we need write code and use parts of the API for deeper analysis? Have we prepared representative environments and representative data to model real-world performance?

  • How might we explore or experiment with the product to examine performance-related risk? How might we stress the product to extremes? How might we starve the product of things that it needs?

It seems to me that examining and exploring these questions will help to guide you to develop a set of ideas for testing performance.

This might help too: https://www.developsense.com/blog/2018/07/exploratory-testing-on-an-api-part-1/

1 Like

I think Michael an important point here - these are the questions that weren’t addressed in the AMA last night. Some of the points you’ve mentioned were addressed and clarified last night. The video will be live later this week for watching.

Metrics are not the first thing I think about when thinking of a Test Strategy. A Test Strategy helps me guide my testing. Drive the choice of testing I do. How do metrics play a part in that?

I have a love/hate relation with metrics. This is because I think it is generally a good idea to have metrics to measure useful and meaningful thing that drive value. That help me make decisions about the way we work or the product we are working on. On the other hand, we should be really careful with metrics. There are many really bad metrics.

What problem are you trying to solve or what insight do you want to gain? Which data could help you make decisions? Try to find metrics that support that. Have a look at the links I provided on “meaningful and bad metrics” to get an idea what could help you.

I cannot imagine that anybody would have a fixed strategy, since we learn new things all the time. So I would expect the Strategy to change too. So yes, it develops. A Strategy starts small. From the start to the finish of a test project or testing a release or even a user story, we start small. How to develop (and decide about) a Strategy is described in question 1.

So a Test Strategy evolves, it grows while we learn & discover more about the product. Based on risks we decide what to test first. For those areas we have more details in the Test Strategy. We go from global to more details. Your Test Strategy is “complete” at the end of your testing.

This is the same as asking “how may organisations or team within an organisation vary?”. A good Test Strategy is product specific so it definitely will vary in different teams/organisations.

Absolutely. And if the project goes as planned, please do so too! The is no way I can imagine where we create the complete Strategy upfront and we stick to it until the end. Also look at my answer for question 9. I think a Test Strategy should evolve from small and general to big and detailed over the curse of the “test project”.

For me a Test Strategy is an evolutionary (mental) model that changes all the time. When I learn new things, when I get new insights, when I see parts of it aren’t working as planned, when I get feedback that the ideas I have aren’t effective or my way of working is inefficient. I share my thoughts and (parts of) my test strategy constantly with many people I talk to or work with. They help me sharpen my ideas.

Why do you want to explain the value? Why don’t you just show the value by example? Tell them about your Test Strategy and show them artefacts to support your story.

You always have a Test Strategy in your head. Maybe you write it down, maybe you don’t. So what is the problem we are talking about here? Do you need to convince them to spend time on creating a document which captures your Strategy? Or do they refuse to help you or spend time on it?

The value of any strategy lies in knowing what to do and how to collect the information needed. The clearer your mental models become, the better you will know what to test, why and how. The strategy will create insight in which information our stakeholders want (missions), what the risks are, what the status of the product is, what actions we need to take to mitigate the risk (test or something else).

A Test Strategy is a set of ideas that guide your testing. A Test Strategy document is something different. Have a look at what Michael Bolton said about that here.

Resources can be found on my blog. Have a look at my Great Resources page. You’ll find a section on Test Strategy with many links that will interest you.

This question was answered in the AMA session. You can find the recording here. The question is answered at 51 minutes and 20 seconds.

First let’s talk about the difference between a test plan and a test strategy. To me a test strategy is a set of ideas (not a document) to guide your testing. This is not a plan. A test plan is the set of ideas that guide your test project. A Test Plan is the sum of logistics (the set of ideas that guide your application of resources to fulfilling the test strategy) and the strategy. Michael Bolton wrote a blog post about this titled “What Should A Test Plan Contain?” almost 12 years ago.

What definition says that? I googled it and found several websites that made me sad. On those pages I read that a test strategy is a high level document. They talk about a strategy being generic requirements for testing or how to test on an organisational level. I am talking about something completely different here! Please remember what Rikard Edgren said in his fabulous tutorial "Test Strategy - Next Level. It is in the combination of WHAT and HOW you find the real strategy. If you separate the WHAT and the HOW, it becomes general and quite useless.

I can not emphasize it enough, A Test Strategy is not a document! Whenever you test anything, you have a Strategy in your head. Most of your Strategy is in your head or in several heads spread over the team and will become clear when you discuss it. Writing things down, making mind maps or other visuals and models only helps you understand and communicate your Strategy better. The trick is to become aware of the Strategy in your head (mental model) and create insight and overview for yourself and your team and the stakeholders.

We have strategies for everything we do, even when we do not write them down or visualise them. Ask yourself, do I need to write it down? Why? To discuss it? To make it better? To communicate it? To report or for accountability purposes?

When you think of how to test a whole project or a separate user story, you think of different things but also about stuff that overlaps. I think it is a good idea to “split” your Strategy up and examine it from different angles: project, features, user stories, releases, etc. Using many perspectives and many models (see my answer for question 1 and question 2), will make your thinking better. Which does not mean it needs separate documents or in some cases even a document at all (a Strategy is not a document remember?).