Do you test in production?

Hey all, the idea of testing in production isnā€™t a new one. However, it seems that there are contexts where itā€™s a good idea, and others where it creates its own risks.

So Iā€™m just interested to get a feel for things. Are you testing in production, and why (not)?

Are you testing in production? Why
  • Yes, we are currently testing in production
  • Yes, but we plan to stop
  • No, but we plan to start
  • No, we do not test in production
0 voters
3 Likes

I feel like I need to explain. Haha! I rarely test in production, but when I do itā€™s because we have a designated production account that does not touch client accounts. So, I know what boundaries Iā€™m working with when testing.

Otherwise, if I had access to client data, I would delightfully decline. I know my propensity to break things and donā€™t want to be able to do something I would regret later.

4 Likes

In Production , only sanity testing is performed and deep testing needs live user account with orders, even if we create one but it would be limited capabilities and access

3 Likes

Most testing for most product features is done in local/test/pre-prod environments. Some features need extra or specific testing in production.
We check in production things weā€™ve just deployed, sometimes briefly, other times more in depth due to data availability or environment specific setup/configurations. One example: to get a similar server setup in the test environment for one sub-system as in prod a company didnā€™t want to pay 200k/year/server, so instead a server with a test setup was used which varied slightly.
We check in production features when we have ab testing or blue-green releases or canary releases. Or when we did demos or alpha/beta testing.
We test in production features fully hidden from the users(with some technical enabling).
We investigate/explore data/logs in production to pinpoint bugs that clients have that are ā€˜silentā€™ to them and bugs not reported by anyone.
We test in production release product versions where thereā€™s multiple departments involved in configuration, administration, data, content management, besides product feature deployment; where thereā€™s no possibility to duplicate the work in test environments.
We ā€˜testā€™(monitor) in production with automated checks, scripts, dashboards, and other tools the stability, availability, reliability, robustness, etcā€¦ of the application over time.
We test/check in production 3rd party system updates, or our systems features linked to 3rd party systems integrations based on a risk based strategy. We had several such surprises where ā€˜hot-fixesā€™ updates have caused problems to our system.

3 Likes

Most companies Iā€™ve been at we needed some production testing or dumping prod data to lower environments. Itā€™s not always the cause but real use data like whatā€™s in prod is a major source of bugs and missing test cases. Staging environments are useful as they should mirror production but this almost always means infrastructure wise. Itā€™s rare Iā€™ve seen high quality real use data in staging unless itā€™s been dumped from prod.

Ideally the system design wouldnā€™t let data get in to bad states. Iā€™ll be honest and say itā€™s rare for developers (I say as a senior dev) to even think about preventing bad states let alone actually pulling it off.

*Test locally, test in dev, in staging, in production. Run it before you push up changes. Pull down PRs and run them.

*There isnā€™t a one size fits all answer unfortunately. Every code base has different testing needs. Every change has different risks and testing needs. Being better requires missing things, feeling the pain, and developing your gut.

4 Likes

So when it comes to testing on production we do it in two ways :

  1. We do sanity testing after a new build is deployed, we have just one user for our project and once the build is deployed we just go to the application, log in with credentials and just load all tabs/screens, and then logout to ensure that the prod build is not getting crashed after the deployment

  2. In this case we donā€™t do testing we use the prod environment only when there is any issue with a specific scenario that we are not able to replicate on the internal environment. So we access the prod environment only to check the logs to find the Root cause. However, this is done mostly by EMs or QA leads/managers.

There are specific protocols setup for those who access prod environment and they have to ensure that every protocol is being followed while the prod environment is accessed by our team.

And we donā€™t do any other testing on prod because any change by our activity may impact the data of real-time users, and it might be possible that compliance issue may also occur.

3 Likes

Tricky. So Iā€™d say no we donā€™t test in production, we do everything we can to gain confidence before deploying to production. Yes our Ops teams will do sanity checking to ensure there are no red flags post deployment, but that to me is common sense - as ops will get the phone calls if it isnā€™t!
Butā€¦we use production at times as part of our testing. When you have realtime data products for multiple clients with multiple formats of data inputs to each of them, one tweak to improve one customer could have an effect on another. So as well as the objective testing of ā€œis it working as we expected?ā€ there is a subjective check against production asking ā€œcould the customer see a negative impact when its deployed?ā€.
So we compare our stagings to prod environments to ensure that the impact of the changes is not going to be seen to have a negative effect on the current production. We could have made the product more accurate with our changes, but if the customer perceives a negative impact we need to be ready to explain why its a positive.

2 Likes

Hello @cassandrahl ,

Yes, I have experience testing in production. Only sanity testing is performed in production and full regression testing on a live user account linked to our account is only possible when we got confirmation from the project manager.

Additionally, in production, issues reported by end users can be tested. I have experience testing in production and have identified critical bugs during the release phase.

Happy Testing! :rocket:
Ramanan

1 Like

Normally as an extension of testing and not as an alternative to other testing we do.

Health checking the small variations between our test and production environment and making sure we have deployed to production okay is the most common one.

We also do A/B testing which needs to be production, alongside analytics, testing users real usage with heatmaps, devices, OSā€™s , and monitoring crash reports, security, traffic volumes etc, maybe even some testing for marketing funnelling. All of this testing contributes to ideas for improvement in addition to catching the odd bug we may have missed.

If there is a good reason and value in production testing then I do not see why not, but Iā€™d be wary if it was seen as a replacement for pre-production testing activities which offer different value.

1 Like

I can certainly relate to that fear of messing up something. Maybe this is a sign that the recovery processes for something like that arenā€™t good enough yet?

2 Likes

Thatā€™s kind of a challenge Iā€™m having on my project, in that all records would be ā€œrealā€ records, and they would affect multiple other dependent teamsā€™ systems too. This means there would be a lot of hoops to jump through to change the environment in such a way that testing in prod would be possibleā€¦ But then the environment no longer has the prod configuration, so what would be the benefit? That benefit vs risk question has been really important for us.

1 Like

I love that question!

1 Like

Oh, I was definitely thinking of testing on prod as an addition, not a replacement, but it is a good point that some places have used it as a replacement. Reading through the comments so far, that doesnā€™t seem to be the case for folks here. I imagine youā€™d need a specific set up in order to enable and warrant testing only on prod.

No, we could fix things if we needed to. Iā€™m just really good at doing unfortunate things :sweat_smile:

1 Like

Shift Right :person_shrugging:

But yea mostly sanity checks APM tests etc.
We also have automated tests to check if configuration is set properly.

It doesnā€™t take days of course, just a few quick tests.

I posted the poll on LinkedIn too, it might help get some responses/insights.

1 Like

Yes, to some extent. Most of the testing is done on dev and staging env, but I sometimes check certain features on production too if I consider them risky or see some value in testing there too.

Itā€™s done on my own prod testing accounts and doesnā€™t affect end users in any way. Cleanup is also performed if needed.

1 Like