For the purpose of this topic, let us not debate about definitions of what either Test Plan or Test Strategy are, the Internet and MoT are full of resources for that. Let’s just say they are what you think they are
Generally, we all know the theory. But in reality, how often do you write those documents, if ever? Why do you write them and for whom? Maybe someone else writes them in your company and you follow that Strategy / execute that Plan?
My personal experience is a mixed bag. In one company there was this huge document of 30+ pages that had all the plans and strategies but in most teams I was the sole QA and there were no documents whatsoever, sometimes an initiative to make one was frowned upon, with reason given “it’s just bureaucracy”.
For those who do write Strategies and Plans, what I want to know do you write them yearly, quaterly, “sprintly”… or once per client/project, etc? Of course it depends on the context, it’s not the same if you’re a test consultancy company or a startup that has a single product.
Thankyou for sharing your thoughts and experiences on this topic.
In my case, the frequency of writing test plans and strategies has varied depending on the project and organized context. Sometimes, I’ve been involved in creating comprehensive documents annually or per project, while in other scenarios, the approach has been more agile, with plans being adapted and updated “sprintly” or as needed.
It’s interesting to hear about the diversity of approaches and how they align with different company cultures and project requirements.
Ive often experienced this. But I always put together at least something of a plan/strategy. Even if it is simply a bullet point list in a Jira or Azure ticket. I always communicate those notes. Even when it is dismissed as busy work because over time, every time, I get buy in. Eventually developers and product owners will realize its an opportunity to negotiate the testing and agree on a “right sized” test as a team and it serves to track the ongoing test work. But it has often taken persistence to get there.
This is an interesting take. I make a lot of such notes / documents, I just never called them Strategy or Plan, even back in the developer days. I believe I lacked the proper terminology. The result was of course there was no clear document structure.
It’s actually quite refreshing and even sometimes amusing to learn new things, phrases, basically expanding one’s vocabulary, only to realise you’ve done this for years, you just didn’t call it as such.
I remember such case being with programming terms like stubs, mocks, spies, tuples, closures… as well as YAGNI principle, KISS principle and plethora of others. I’ve done all of those, just didn’t know they had a special fancy name
I feel Test Strategy and Test Plan are such terms from QA world, tnx for triggering me to go down the memory lane
People often think of a “Test Plan” as a massive thing, owing back to the waterfall method in which large tomes of documentation were produced before any development got started. It doesnt have to be a massive thing. A simple articulation of “Must test, should test, could test, wont test” things for a given delivery of code, no matter the size, is all it need be. Its an opportinity for dev and product to understand that portion of the work and to have input, if they wish to. “Oh no, I dont care about that thing, it can be a ‘could test’ item but isnt a priority” says the PM.
Great question, over the course of 7 years I got to write once a proper Test plan document over 20 pages. This was required because the project was in waterfall model and the company was heavy in documentation. It was nerve racking because of the size of document, ownership and all the questions I would get after it.
But in all other companies, I have seen test doc or automation strategy doc not more than 2 pages with just bullet points.
I’ve been at my current company for almost seven years, at senior position for most of that time, and I can count them on one hand.
At best I would get a comment or question from someone else in testing. They would point out places that require clarification or things I did not think about. These are genuinely helpful. At the same time, we could have achieved the same results, while increasing social cohesion, if company had different culture around collaboration and testing in general. I could spend 2 hours at generating test ideas, everyone else could spend 15 minutes, then we could get at video call and bounce ideas off each other.
At worst, test plan would not be read by anyone. I am pretty sure this is the fate of some documents that I have created in my previous company.
In my current project we mostly follow scrum ceremonies - we do backlog refinements, write acceptance criteria as a group, testers are in all the same meetings as developers etc. We also have CI in place, and try to cover new features in tests as they are introduced. Since new features are broken down to small chunks, a culture of unit testing and writing acceptance criteria mostly solves the same problems that test plans would have.
Im just entering a similar situation in which the org has run without any QA resources for some time and very poor or ill-fit QA resources prior to that. BUT they are very good at engineering testing, as in developers test and write tests as they feel is necessary. Its worked…enough, for some time but now they are feeling the pinch of scaling that and the growing tech debt as the gaps in more cohesive test activities, process and planning are being felt. So Im tasked with “fixing” all of that. One of the things I am pushing forward is retaining that current sense of responsibility on the part of engineers. Continuing to empower them to take responsibility for the quality and to expand the test effort to include more people as needed. The organization does a good job of articulating more involved efforts in brief design and business rule documentation. This makes it easy for me to create equally brief test plans. But here is the point - even if no one looks at that test plan despite my making it available - It serves me in organizing my thoughts and effort as well as estimating effort over the smaller incremental changes.
Thanks for articulating more of the iterative releases and how the docs play a role in your process!
I try to create one for each feature and share with the team.
I find it useful because it forces me to think about the feature, record and organize my thoughts, document what I would want to test and even include notes on how to tests.
Also, it’s useful when I revisit the feature after a while.
We have a shiny, corporate test plan template that covers significant releases. Strategy gets covered elsewhere (sometimes in a task Jira) and quite informally. It’s a valuable thing to have. For example, we might agree we need an engineer to get a harness they’ve written to expose an API we can use when testing a new feature.
I have reservations about doing test strategy in a formal sense because there isn’t much agreement as to what a test strategy is and contains. The test plan comes with its own ISO standard.
We do outsourcing work for our clients: public organizations and private companies. There are usually contractual requirements for test strategy and test plan documents on both when to deliver and the content of the documents.
Usually a strategy document pr. program/project, and a test plan pr. testing activity. Things like functional testing, disaster recovery tests, and penetration tests are usually in scope.
thank you all for sharing the different approaches
Tnx for sharing your feedback and tnx for coming back here after so much time
And yes that’s a valid thinking in a lot of companies. For example when there’s a product that’s been on a market for years and there’s a team and there’s an established process, there’s really not much need for any such document.
Exception to the case could be when there’s major rewrite in the pipeline, or a new major version in the project that could have some breaking changes, then in my opinion it would be good to have some plan in advance
I was always puzzled how this use case is functioning. What I want to say is that for a good strategy one needs to have great understanding of the context, that includes knowledge of the company down to internal workings of team(s) and project(s) that strategy is written for… what kind of team members they have? what’s their tech stack? what are their strengths? weaknesses? what’s the company vision? etc.
I guess there’s lots of back-and-forth sync between the consultants and the payee?
oh yes. We often have a dedicated person in charge of the test coordination internally and externally. We call it a Test manager, it’s similar to a specialized project manager.
Unfortunately, the test strategy documents are not strategic as you mention, but often a formal description of a way of working and tool setup. There is even an ISO standard for these documents.
My personal opinion is that they are “empty calories” that seem fulfilling but don’t actually help the “payee” solve their core problems.
…and usually only myself and one other person reads them ever, so the last few years, I write these docs as if they are notes to future me. The test report itself is more a formal thing, people will actually look at that. But these are all just snapshots. Test plans are not a “output” on the production line, so making them look pretty is really not worth your time.
I used maintain test cases on testrail but owing to the cumbersome task of updating the old cases in case of changes to software (tech startup which is ever so changing) and the fact that testrail is very buggy, I decided to drop this practice.
Now though, I’ve resorted to creating test activity charters that’s just 2 columns with 1 containing the test activities and the other listing down the test oracles that cover each activity.
I do this only for medium to larger impact stories (i.e. ones which have a lot of use cases and resulting test cases to be covered)
This way I’m not bound by a format and I can do a free writing brainstorm session as I go along creating the actual document.
Can you provide a simple example? Do you write this docs in a task management tool? By the first look I see this as a QA task for a specific story. For example, in my team we opted for creating two subtasks on each story that we as QA can test:
Test Case creation & update
QA Testing
First subtask is assigned to person who, as title says, checks story requirements and decides if any tests need updating or if additional ones need to be created from scratch. This should in ideal case be done before devs start working on the task, or before the start of testing at the latest.
Second subtask is assigned once the testing is started. It can also sometimes be ommited in favor of some scrum board column status but we like it this way so the task is assigned to actual person and then you can check what that person is doing, same as the devs.
I’ve had some strongly held yet easily changed beliefs around writing these documents.
In the past we’d have detailed documents written per project where we articulate everything, from what stories will be tested (all of them), to the environment including specific test machines. These would be reviewed, updated then ultimately ignored. This was abandoned. When told to pick this up, adding in different automated tests and specifying their pipelines, we groaned and would copy-paste a document, modifying as necessary. Waste of time. Instead I wanted our Mural showing a “ways of working” to be all we ever had. My strongly held belief was these documents were a waste of time.
Lately I’ve changed my mind. I’ve written a strategy for our overall multi-year project. It is higher level. It is about our aspirations and what I’ll be challenging the teams on. For each epic I am pushing for teams to collaborate to write something. Instead of being a document, we have a Mural template to fill out. The point here isn’t to have a document that will never be read to outline what we’ll do. It is to get the team to talk about the testing needs. To think about environment, NFR and potential challenges at the start.
On a per story level, I like flexibility. There’s always value in thinking, and putting words in place, about what you need to do (I love a bulleted list) but writing something formal isn’t for me. If the story has notable risk (e.g. area known for regressions) or I don’t understand the story (or potential impacts) then I’ll ask for a once over… or even better, a quick chat.
