Which Environments Should We Be Testing In? CI/CD

Processes
#1

Hi guys,

We currently have 4 environments for our new web application which is currently in the beta stage. Development, QA, Staging and Production.

My question is; what type of tests should we/would you be carrying out in each environment?

Both myself and the engineering department as a whole are new to CI/CD, so we’re trying to work this out together.

At the moment:

Dev - only sanity checks are carried out by developers here
QA - this is where we check all functional code changes. Regression will be carried out here.
Staging - performance tests will be carried out here, since it is a mirror of our Production environment (ideally we would create and then tear down a temporary environment for this). This is where Product Management will ‘sign off’ any features tested by us.
Production - we’ll be testing SQL database updates (that don’t require a code change) here. These tests will be non-intrusive. Should we be carrying out smoke tests here? To make sure the main pages are loading etc?

We’ve only recently created these environments(as I’ve mentioned, we’re at the beta stage) so any hints/tips and useful articles would be really helpful.

Thanks guys.

1 Like
#2

Hi, and what is your git flow? To me its not so much important what env you test in but if testing envs follows your development in most effective way possible.

In my company we basically use the same envs with few changes. QA is more production like than other environments, staging is for final light weight check before signing off release.

#3

Thanks for your reply!

By git flow, do you mean where do code changes get deployed to first?

Dev > QA > Staging > Production

  1. Dev is for developers to sanity check their changes, make sure it hasn’t ‘broken’ anything major. Devs approve their changes to move into…
  2. QA is where we test the changes functionally. We then sign off and the changes are automatically deployed to…
  3. Staging which is where product management ‘sign off’ the changes (either demonstrated by us or quickly checked by them) and then these are signed off by Product Management to move onto…
  4. Production. The environment the clients will be using.

The reason I ask this question is because we were getting some push back into the types of tests we were carrying out in each environment. E.g. why would you test data deployment on production and not on QA? Our answer was that code changes and data changes should be tested separately.

We’d also like to perform smoke tests here (e.g. are apis responding, can all UI pages be loaded), at least whilst our deployment methods are in their early stages.

We’re also getting push back on performing performance tests on Staging. My argument for doing this is that it is an exact replica of our production environment in terms of resources. Of course, we have to be responsible and carry out these tests outwith working hours.

A bit of a ramble there… but hopefully you guys get the picture?

#4

An excellent question, and one I often struggle with. The flow/environment types you have are similar to the deployment pipeline we typically have.

I would run my smoke, API and UI tests against QA. In that order, and block if any fail. You want to move up the testing pyramid - fastest tests first, so you can fail faster.

Data tests are more interesting. It depends what you mean - we had an application where we were testing data migration from an older system to a new system. Our prod deployment service included that data migration and transformation (took a long time). After we’d migrated the data, we ran some quick checks against it on the prod environment. I’m not entirely sure whether that was the best way to do it.

For performance tests, I still haven’t decided. You could run them on staging, given that performance testing times are clearly signposted (and people may experience latency). You could also spin up a new temporary environment - if you’re using infrastructure as code this should be relatively doable. Generally our staging env isn’t as well resourced as our prod environment, so it’s not suitable for perf testing, so we spin up a new, temporary environment. If you’re blue/greening, you’re paying for two envs anyway, so why not use the one you’re not using?

This stuff is contextual, so I don’t think it’s a clear-cut solution, but hopefully I’ve provided some insight?

#5

Thanks for your reply, very useful.

I have a couple of questions - why would we not smoke test against production? Make sure that the environment is still alive and all APIs are up and running.

I agree that the majority of testing should be done against the QA environment, and maybe it’s because I don’t trust our current deployment methods enough that I want to test against production?

As for the data tests - data would be deployed to Production but only turned on for a Test/Demo client. This would allow us to carry out database checks and ensure the data update hasn’t broken any functionality. As soon as these tests pass, we’ll ‘flick a switch’ and the latest version of the database will be available for all clients.

Our thinking is that we want to keep functional code changes and database testing separate.

In regards to performance testing - I’d agree that a temporary environment would be best. Unfortunately we don’t have that as an option immediately, but it’s definitely something we’ll be aiming for.

#6

You can run smoke against production if you want, too. That would give you some certainty that the release process was effective. You run the risk of injecting test data into production, though. It depends how you do test data - we randomly generate data for each test, which makes the tests idempotent.

As far as ‘testing’ on production, we tend to do monitoring and healthchecks over functional tests. I’ve been doing a lot of infra testing recently, and you can set up a Goss healthcheck server, if you’re curious about tooling: https://github.com/aelsabbahy/goss

If you’re running a 12 factor app then your environments should at parity. If not then I would definitely test production, as you technically haven’t tested your whole application.

I think your data tests sound very similar to ours. Ours were more ‘data migration’ tests. Once we’d migrated from the legacy system to the new system, we stopped running them. I’m not sure what happened with regards to database testing afterwards… if indeed, we did any…

Yeah, a temporary environment is ideal, but ideal isn’t always possible! It’s all about improving…

#7

I think what I’m getting from your reply, and from what I’m reading online, is that there is no right or wrong answer. Especially not at the early stage of us implementing this deployment process.

Thanks!