What are the different approaches/strategies do software testers are opting for nowadays while driving their testing "Testing in production" from observability perspective

Most people convey that “There is no, has never been, and will be no environment that looks just like production”.

Yes, this is true, and approaching testing in production also reduces a cost, resources, time, and effort perspective.

  • How you are using observability in your day to day “testing in production” ?

  • What the good balance do you come across between testing left and testing right ? What you have tried so far?

  • My experience so far with production testing is that I have driven myself to do the sanity testing, A/B testing, setting up, maintaining, monitoring the important flows, monitoring SLA, testing via feature flags, tackling testdata etc but nothing as such having any specific strategy for production testing while eliminating the risk, and have a mitigation plan to have the agreement of different stakeholder. I also see most of the company have a policy not to perform any testing in production. I did not do much for observatility prespective.

2 Likes

Testing in production is not for everyone and will always mean different things for people.

  1. For example: Test automation tools can never cover all authentication workflows in automation without modifying the security criteria slightly, so that’s the biggest reason you don’t “automate” in production. Any manual testing you do in production pales in comparison to what you can accomplish with automation, so you have to choose specific cases to cover in production on ad-hoc bases.
  2. Production testing is not actually testing the code the devs are writing right now, it’s too late. Thinking of ways to save money by testing in production is not the test engineers job at all, it’s cheaper to have 3 environments and catch bugs than to have 2 environments.
  3. Production is often owned by a different team, and unless you use a A/B swap-over system for production, it’s likely to be so different to develop and staging internally, that production testing is not going to find product defects, but instead find environment defects you might not care about right now.

So yes, you need a balance. How you balance will depend very much on policy specific and architecture and deployment specific sources of friction.

5 Likes

Thank you for your suggestion! Appreciate it!