We currently have 4 environments for our new web application which is currently in the beta stage. Development, QA, Staging and Production.
My question is; what type of tests should we/would you be carrying out in each environment?
Both myself and the engineering department as a whole are new to CI/CD, so weāre trying to work this out together.
At the moment:
Dev - only sanity checks are carried out by developers here
QA - this is where we check all functional code changes. Regression will be carried out here.
Staging - performance tests will be carried out here, since it is a mirror of our Production environment (ideally we would create and then tear down a temporary environment for this). This is where Product Management will āsign offā any features tested by us.
Production - weāll be testing SQL database updates (that donāt require a code change) here. These tests will be non-intrusive. Should we be carrying out smoke tests here? To make sure the main pages are loading etc?
Weāve only recently created these environments(as Iāve mentioned, weāre at the beta stage) so any hints/tips and useful articles would be really helpful.
Hi, and what is your git flow? To me its not so much important what env you test in but if testing envs follows your development in most effective way possible.
In my company we basically use the same envs with few changes. QA is more production like than other environments, staging is for final light weight check before signing off release.
By git flow, do you mean where do code changes get deployed to first?
Dev > QA > Staging > Production
Dev is for developers to sanity check their changes, make sure it hasnāt ābrokenā anything major. Devs approve their changes to move intoā¦
QA is where we test the changes functionally. We then sign off and the changes are automatically deployed toā¦
Staging which is where product management āsign offā the changes (either demonstrated by us or quickly checked by them) and then these are signed off by Product Management to move ontoā¦
Production. The environment the clients will be using.
The reason I ask this question is because we were getting some push back into the types of tests we were carrying out in each environment. E.g. why would you test data deployment on production and not on QA? Our answer was that code changes and data changes should be tested separately.
Weād also like to perform smoke tests here (e.g. are apis responding, can all UI pages be loaded), at least whilst our deployment methods are in their early stages.
Weāre also getting push back on performing performance tests on Staging. My argument for doing this is that it is an exact replica of our production environment in terms of resources. Of course, we have to be responsible and carry out these tests outwith working hours.
A bit of a ramble thereā¦ but hopefully you guys get the picture?
An excellent question, and one I often struggle with. The flow/environment types you have are similar to the deployment pipeline we typically have.
I would run my smoke, API and UI tests against QA. In that order, and block if any fail. You want to move up the testing pyramid - fastest tests first, so you can fail faster.
Data tests are more interesting. It depends what you mean - we had an application where we were testing data migration from an older system to a new system. Our prod deployment service included that data migration and transformation (took a long time). After weād migrated the data, we ran some quick checks against it on the prod environment. Iām not entirely sure whether that was the best way to do it.
For performance tests, I still havenāt decided. You could run them on staging, given that performance testing times are clearly signposted (and people may experience latency). You could also spin up a new temporary environment - if youāre using infrastructure as code this should be relatively doable. Generally our staging env isnāt as well resourced as our prod environment, so itās not suitable for perf testing, so we spin up a new, temporary environment. If youāre blue/greening, youāre paying for two envs anyway, so why not use the one youāre not using?
This stuff is contextual, so I donāt think itās a clear-cut solution, but hopefully Iāve provided some insight?
I have a couple of questions - why would we not smoke test against production? Make sure that the environment is still alive and all APIs are up and running.
I agree that the majority of testing should be done against the QA environment, and maybe itās because I donāt trust our current deployment methods enough that I want to test against production?
As for the data tests - data would be deployed to Production but only turned on for a Test/Demo client. This would allow us to carry out database checks and ensure the data update hasnāt broken any functionality. As soon as these tests pass, weāll āflick a switchā and the latest version of the database will be available for all clients.
Our thinking is that we want to keep functional code changes and database testing separate.
In regards to performance testing - Iād agree that a temporary environment would be best. Unfortunately we donāt have that as an option immediately, but itās definitely something weāll be aiming for.
You can run smoke against production if you want, too. That would give you some certainty that the release process was effective. You run the risk of injecting test data into production, though. It depends how you do test data - we randomly generate data for each test, which makes the tests idempotent.
As far as ātestingā on production, we tend to do monitoring and healthchecks over functional tests. Iāve been doing a lot of infra testing recently, and you can set up a Goss healthcheck server, if youāre curious about tooling: https://github.com/aelsabbahy/goss
If youāre running a 12 factor app then your environments should at parity. If not then I would definitely test production, as you technically havenāt tested your whole application.
I think your data tests sound very similar to ours. Ours were more ādata migrationā tests. Once weād migrated from the legacy system to the new system, we stopped running them. Iām not sure what happened with regards to database testing afterwardsā¦ if indeed, we did anyā¦
Yeah, a temporary environment is ideal, but ideal isnāt always possible! Itās all about improvingā¦
I think what Iām getting from your reply, and from what Iām reading online, is that there is no right or wrong answer. Especially not at the early stage of us implementing this deployment process.
CI/CD stands for Continuous Integration and Continuous Deployment/Delivery . A CI/CD environment is a type of production environment in which ongoing automation and monitoring are implemented to improve and expedite development processes for integration and testing, and deployment and delivery. Its aims to solve those issues that development teams are facing with when integrating and merging new code additions.
Now, the time-consuming merge days are replaced with ongoing automated data migration testing that keeps code updates flowing through the production cycle. In a CI/CD environment, development and operations collaborate closely together with an agile approach as new code additions are moved through a CI/CD pipeline designed to continuously integrate and deploy new product fixes, updates, and features.
CI is an automation process within development that continuously builds, tests, and merges new code changes. This allows developers to quickly write and merge new code to a single, shared ātrunkā instead of having to wait to manually merge many individual branches of code
However, CD refers to the automated process of taking the validated code additions from the CI processes, and releasing it to a shared repository (such as GitHub). The goal is to maintain an up-to-date codebase where new updates and features could be easily deployed to a live production environment.
Here another question comes how can we get started with implementing CI/CD at our organization? So, considering the level of well-designed automation required for an effective CI/CD pipeline, there is quite a bit of upfront labor that must be done first. Weāll need to start with establishing an intelligent system of CI designed for our specific development ecosystem. Our CI/CD pipeline relies on a high degree of ongoing automated testing and monitoring at the CI level in order to effectively deliver and deploy new code updates without breaking the application.
Also, there are several tools available to help get started with CI/CD as well as to improve our existing processes. CI/CD tools like Jenkins are great for automating testing/integration, delivery, and deployment.