Hey all, the idea of testing in production isnāt a new one. However, it seems that there are contexts where itās a good idea, and others where it creates its own risks.
So Iām just interested to get a feel for things. Are you testing in production, and why (not)?
I feel like I need to explain. Haha! I rarely test in production, but when I do itās because we have a designated production account that does not touch client accounts. So, I know what boundaries Iām working with when testing.
Otherwise, if I had access to client data, I would delightfully decline. I know my propensity to break things and donāt want to be able to do something I would regret later.
In Production , only sanity testing is performed and deep testing needs live user account with orders, even if we create one but it would be limited capabilities and access
Most testing for most product features is done in local/test/pre-prod environments. Some features need extra or specific testing in production.
We check in production things weāve just deployed, sometimes briefly, other times more in depth due to data availability or environment specific setup/configurations. One example: to get a similar server setup in the test environment for one sub-system as in prod a company didnāt want to pay 200k/year/server, so instead a server with a test setup was used which varied slightly.
We check in production features when we have ab testing or blue-green releases or canary releases. Or when we did demos or alpha/beta testing.
We test in production features fully hidden from the users(with some technical enabling).
We investigate/explore data/logs in production to pinpoint bugs that clients have that are āsilentā to them and bugs not reported by anyone.
We test in production release product versions where thereās multiple departments involved in configuration, administration, data, content management, besides product feature deployment; where thereās no possibility to duplicate the work in test environments.
We ātestā(monitor) in production with automated checks, scripts, dashboards, and other tools the stability, availability, reliability, robustness, etcā¦ of the application over time.
We test/check in production 3rd party system updates, or our systems features linked to 3rd party systems integrations based on a risk based strategy. We had several such surprises where āhot-fixesā updates have caused problems to our system.
Most companies Iāve been at we needed some production testing or dumping prod data to lower environments. Itās not always the cause but real use data like whatās in prod is a major source of bugs and missing test cases. Staging environments are useful as they should mirror production but this almost always means infrastructure wise. Itās rare Iāve seen high quality real use data in staging unless itās been dumped from prod.
Ideally the system design wouldnāt let data get in to bad states. Iāll be honest and say itās rare for developers (I say as a senior dev) to even think about preventing bad states let alone actually pulling it off.
*Test locally, test in dev, in staging, in production. Run it before you push up changes. Pull down PRs and run them.
*There isnāt a one size fits all answer unfortunately. Every code base has different testing needs. Every change has different risks and testing needs. Being better requires missing things, feeling the pain, and developing your gut.
So when it comes to testing on production we do it in two ways :
We do sanity testing after a new build is deployed, we have just one user for our project and once the build is deployed we just go to the application, log in with credentials and just load all tabs/screens, and then logout to ensure that the prod build is not getting crashed after the deployment
In this case we donāt do testing we use the prod environment only when there is any issue with a specific scenario that we are not able to replicate on the internal environment. So we access the prod environment only to check the logs to find the Root cause. However, this is done mostly by EMs or QA leads/managers.
There are specific protocols setup for those who access prod environment and they have to ensure that every protocol is being followed while the prod environment is accessed by our team.
And we donāt do any other testing on prod because any change by our activity may impact the data of real-time users, and it might be possible that compliance issue may also occur.
Tricky. So Iād say no we donāt test in production, we do everything we can to gain confidence before deploying to production. Yes our Ops teams will do sanity checking to ensure there are no red flags post deployment, but that to me is common sense - as ops will get the phone calls if it isnāt!
Butā¦we use production at times as part of our testing. When you have realtime data products for multiple clients with multiple formats of data inputs to each of them, one tweak to improve one customer could have an effect on another. So as well as the objective testing of āis it working as we expected?ā there is a subjective check against production asking ācould the customer see a negative impact when its deployed?ā.
So we compare our stagings to prod environments to ensure that the impact of the changes is not going to be seen to have a negative effect on the current production. We could have made the product more accurate with our changes, but if the customer perceives a negative impact we need to be ready to explain why its a positive.
Hello @cassandrahl ,
Yes, I have experience testing in production. Only sanity testing is performed in production and full regression testing on a live user account linked to our account is only possible when we got confirmation from the project manager.
Additionally, in production, issues reported by end users can be tested. I have experience testing in production and have identified critical bugs during the release phase.
Happy Testing! 
Ramanan
Normally as an extension of testing and not as an alternative to other testing we do.
Health checking the small variations between our test and production environment and making sure we have deployed to production okay is the most common one.
We also do A/B testing which needs to be production, alongside analytics, testing users real usage with heatmaps, devices, OSās , and monitoring crash reports, security, traffic volumes etc, maybe even some testing for marketing funnelling. All of this testing contributes to ideas for improvement in addition to catching the odd bug we may have missed.
If there is a good reason and value in production testing then I do not see why not, but Iād be wary if it was seen as a replacement for pre-production testing activities which offer different value.
I can certainly relate to that fear of messing up something. Maybe this is a sign that the recovery processes for something like that arenāt good enough yet?
Thatās kind of a challenge Iām having on my project, in that all records would be ārealā records, and they would affect multiple other dependent teamsā systems too. This means there would be a lot of hoops to jump through to change the environment in such a way that testing in prod would be possibleā¦ But then the environment no longer has the prod configuration, so what would be the benefit? That benefit vs risk question has been really important for us.
I love that question!
Oh, I was definitely thinking of testing on prod as an addition, not a replacement, but it is a good point that some places have used it as a replacement. Reading through the comments so far, that doesnāt seem to be the case for folks here. I imagine youād need a specific set up in order to enable and warrant testing only on prod.
No, we could fix things if we needed to. Iām just really good at doing unfortunate things 
Shift Right 
But yea mostly sanity checks APM tests etc.
We also have automated tests to check if configuration is set properly.
It doesnāt take days of course, just a few quick tests.
I posted the poll on LinkedIn too, it might help get some responses/insights.
Yes, to some extent. Most of the testing is done on dev and staging env, but I sometimes check certain features on production too if I consider them risky or see some value in testing there too.
Itās done on my own prod testing accounts and doesnāt affect end users in any way. Cleanup is also performed if needed.