What are your thoughts on “Post-Release Test Strategy”? How can I implement it in my company?
Thinking about this question is a great thought exercise to figure out everything you would do to vet your freshly deployed production environment. You can then take the list you come up with and think about how you could get the same confidence before actually having deployed your code. At Twitter, this thought exercise lead to the creation of Diffy to answer the first few questions and since then we have been building Isotope to answer all the remaining ones. Hope these links help.
+1 to Diffy. I hadn’t heard of Isotope, I’m gonna check it out!
If your company uses Datadog then I recommend checking out Datadog Synthetics. With it you can write a test that repeats on a configured interval and alerts when the test fails. Very useful for monitoring and can be tied into rolling back releases if desired.
Here are my thoughts:
- Do you already test before release on some pre-prod environment?
- Are most releases tested in production only?
- Are the testers in charge of release management or in the release core team?
- Who’s owning the product and pays for the product increment release?
- Do you have access to production monitoring and logs?
- Are you/the testers aware of what is released and when?
- Is there any production process for questions, incidents, feedback of different users/stakeholders/departments reps that are already used in the company? or do you need to agree on/establish one?
- What things do you already have/being used?
- What are the main problems that you hope to fix with this strategy?
- Is the release process that’s under testing or is it the product?
Different products, different stakeholders, different risks and problems, different budgets, different resource allocation, different project stages are impacting what you could be doing for the testing and release strategy.
In some companies the release management might have issues - so the test strategy could involve evaluating where the problems are occurring, why, how often, could they be treated in some way, how about avoided, is it human or process or uncontrolled factor related…
Some companies have instability, special rules/firewalls, etc of production systems and not everything could be tested reliably on pre-prod environment. In this case you have to do some testing to evaluate environments and gaps and how you could deal with those.
In some places the release is done by someone or some department, that doesn’t share the changes or the fact that a release has been made. Deal with that problem - try to convene a way for people to communicate and inform and evaluate the risks…