Hi everyone, thank you to all those that came along to the Masterclass session. I hope you managed to take away something useful from it.
So to dig into the unanswered questions then!
Questions to be answered
1. How does this manifest itself in agile contexts? Iâm not sure why but your advice sounds familiar for ye olde test documentation heavy dev approaches - how does it apply to more modern ones?
Absolutely! Being Agile doesnât mean not having any documentation. To help frame a response the manifesto says they value:
Working software over comprehensive documentation
https://agilemanifesto.org/
I agree with this. A team will not read a Test Strategy that is overly verbose and full of unnecessary details.
In the talk, I refer to this by the 100-page strategy example. Who reads that much Test Strategy? I donât!
But so often teams use Agile methodology as a justification to have a Strategy that is sparsely detailed. It misses out on so much information. The why, where, what, when, who and how. As well as the context the strategy operates in.
The key is balance. We donât always need to go overboard and write very long document-heavy strategies. Instead, we should consider our project context and reader. Tailoring to what is expected. That way we can avoid older style very heavy approaches and architect a test strategy that is more modern.
Also, remember experimentation. It doesnât have to always be a formal document. Combine text with other mediums, pictures, video, audio. Experiment with what works for your team.
2. For large ongoing projects/programs, Test Strategy is more a âWorking documentâ. Who is responsible for updating it - it is quite obvious, but who should contribute here?
It varies depending on the scope of the strategy. For an individual team, I would suggest everyone on the team is responsible. The whole team has a responsibility to deliver at acceptable quality levels. Use pairing, rotate pairs perhaps have different pairs complete different sections. Then review at the end as a team for consistency.
If the scope is wider - perhaps a whole organisation. Form a sensibly sized group of individuals to work on and keep the strategy relevant by responding to that organisations changing context over time. Just be careful of the fine line of deliberate considered design and design by committee!
3. Do we write a test strategy for project ? for feature? for epic? or what is the level at where we set a test strategy?
It is up to you! What makes sense for your organisation? I would suggest most teams would benefit from a carefully architected test strategy which they all had a part in developing (see answer above). You might create a strategy for a feature/epic but it is rare in my experience. This is where the line gets blurred and test plans might be better used. Adding to the ticket or somewhere else a short summary of the testing you, the developer or pair plan to do which builds a very specific case based on the strategy.
Some teams like this level of detail - others find it restrictive, bordering on micromanagement - those teams might prefer a Three Amigos session instead which is almost the same thing but often gets better engagement: What are the Three Amigos in Agile? | Agile Alliance
4. If you would need to limit your test strategy to only 1 slide. What would go there?
Why do you want to limit it to one slide? Is it to keep your teamâs attention perhaps? I would try to not limit myself to an arbitrary one slide and instead focus on creating something that is right-sized for the context and reader(s) I am catering for.
But if I had to stay to one slide. I would focus on summarising the project risks and how they will be addressed.
5. We had a test strategy in our company but we are leveling up our QA and we want to revamp our processes.How can we start fresh?
Talk to people in your organisation about the current strategy. Get feedback about what different individuals like and dislike. Then for the whole company put together a little group (see question two for detail) to discuss feedback and plan a way forward.
Then iterate on the strategy. Create an initial version - present it to your organisation and respond to feedback. Build consensus. Your group doesnât have to take on board every suggestion but it is amazing how much more responsive individuals are when you help them feel heard.
6. How do we know whether our strategy is good or bad?
There are two approaches - the first is introducing carefully selected metrics that tie in with your strategy guiding principles. You could use these to track implementation success over time.
But secondly⌠do they read it! Do you have a method right now to track how many times individuals interact with the document like how Confluence has viewer statistics? Can you talk to the team(s) to see if they actually have read and understood the strategy taking on board the actions and so on.
Out of the two, I prefer the second - seeing how much teams interact with it. Metrics have a surrogation risk
Surrogation is a psychological phenomenon found in business practices whereby a measure of a construct of interest evolves to replace that construct.
Or in other words, the metrics become more important than the objectives they track. So I would be careful introducing them.
7. Could you mention various ways of publishing these strategies. For example, Confluence, etc. What do others use?
Yes of course. There are many different formats you can use:
- Confluence
- Word Documents
- PDF
- HTML
- Markdown
It really depends on the style and formality of your document. PDF and Word are well suited to more formal contexts - where perhaps a formal review and acceptance process needs to happen for publication of the strategy.
HTML and Confluence are interesting options. You can embed interactive content easily which can make your strategy more engaging and accessible. Not everyone learns best from reading text. Some (like myself) prefer video or audio. So for example you could insert lightning talks from your favourite conference into your strategy to further reinforce the message.
Markdown is great because of its lightweight styling options and developers like it. You can publish your strategy in a Git repository easily with this method. That gives you excellent versioning options and a natural way for developers to contribute.
8. Do organizations typically have explicitly different strategies for regression testing, bugfix testing, and new feature testing? Or is that overkill?
I havenât seen much of it myself but I wouldnât want to rule it out. Perhaps they operate in a very high risk (e.g. risk to life) environment?
Your example sounds like it could be at real risk of strategies drifting from each other. Causing mixed messages on approach and detracting value. Making teams less likely to read.
Test plans (see question 3) might help replace some of these strategies and realign the main strategy to a more unified Diagnosis.
9. Whatâs the best way to get people to confirm that they have read and understood the strategy? We have a process where we want sign off of the strategy from all stakeholders and we are forever chasing them for this.
I empathise this is something I struggle with too. There are three tactics I use:
- Limit the number of reviewers - trying to avoid a design by committee situation (which it sounds like you might be facing). Does every stakeholder need to sign off? Is there a trust issue in the organisation - why?
- Stakeholder management. Unfortunately, this starts to stray more into how you manage relationships during this review process. If I know there is a particular stakeholder who will have lots of feedback to manage then I will try to organise a separate 121 to try to address some of it ahead of the whole group. To try to manage the amount of feedback that is being dealt with at once.
- Highlight that a Strategy is not a fixed document. They can and should change. Adapting to the changing context they operate in. Is what we have good enough for this version? Can the stakeholderâs feedback be addressed in a future version?
If the stakeholders are really not engaging then you need to ask yourself if they are the right person to review perhaps someone else should be nominated. Or perhaps a deadline should be set for feedback⌠but this is really a last resort.
Perhaps Tom can reply to some of the remaining questions. Until then here are some links to similar club threads: