Test environments - the more the merrier?

[laszlocloud (Laszlo)]

Hi Testing Community!

For a long time I wanted to get the opinion of the test folk (aka You ) about the subject above: Test environments - the more the merrier?

Having a Devops background I certainly find my intellectual fix in git branching models and automation that allow launching new test environments on demand, even per feature branch. But I’m not sure if this is considered important for test people who actually do the tests, manual or automated.

So how many is ideal?
Is the Dev, Staging, Production just enough?
How many test envs do you have?
Is it a concern for you?

Appreciate any response!

Best regards,
Laszlo

[utchbe (Butch Mayhew)]

I think it will depend on the product and teams you’re working on. For our flagship product where we could have 2-3 development teams working on multiple features simultaneously, we have six feature environments, a hub environment (beta), a regression environment (staging), and code that matches production (master). Our branching strategy revolves around a release train (similar to Spotify) where whatever is in the Beta Branch at Code Cutoff will be released to Production within 9 days. We use the hub environment (beta) as the branch where all new feature branches are created from and kept in sync with. We have all of our feature environments named after Star Wars characters. Very rarely do we are out of environments. Our Tech Ops team (who handles the QA infrastructure has everything scripted out using Chef, so we could add additional environments if needed but there hasn’t been a need. As far as automation we have built our suite in a way that the automation is run against all of our test environments nightly. If you have any questions or would like to know more, let me know.

For other products with single teams working on them, 2-3 environments worked out just fine.

My answers:
So how many is ideal? It depends
Is the Dev, Staging, Production just enough? For some teams/projects
How many test envs do you have? 11 internal for QA for our flagship product
Is it a concern for you? We are happy

Butch

[teemuvesala (Teemu)]

I’m also DevOps guy (but also been the developer, and quality/testing professional). And views are my own. If the application is built wisely, you have some kind of minimum number of test environments (e.g. dev & staging for the official tests), but upper limit is just the amount of money you can use.

Rules
First: Use the cloud (AWS/Azure/what ever fits to you the best).
Second: Automate your CI/CD pipeline so that it doesn’t care where you are going to deploy.
Third: Automate also some test data baseline setup.
Fourth: Let anyone (tester/dev) set up the environment when she needs it.
Fifth: Automate at least the smoke tests which takes less than 10 minutes

My philosophy is that I try to encourage also developers to do the testing for their code. And doing that should be as simple as possible. I don’t want to be the bottle neck for their activity. So I rather encourage them to push the changes to own branch. Hit the “build & deploy” button, take the coffee break, come pack and they have test environment built.

This is idealistic view. At my next role I’ll encourage the other developers for this kind of behavior. Automate as much possible. (I’ll actually be the developer again… )

Teemu

[laszlocloud (Laszlo)]

It’s really interesting to read about real life setups. Appreciate your answers.

I’m definitely sharing your views @teemuvesala to automate things to the level where environments can be launched on-demand, and very happy that you mentioned the test data baseline. I sort of failed to establish that in my previous job…

I guess what I’m trying to figure out with this thread is how much of an “idealistic view” is what you described.

Because I think the tooling exists to achieve all the things you highlighted, I just don’t know where the world is right now. We could talk in ideals, that on-demand is more cost-effective than having a set of fixed environments, or other typical automation and CI/CD panels… I just don’t know how the benefits are generally perceived. Plus I don’t have the hard facts on the TCO. Automation definitely comes with an upfront investment.

I know you wrote that you are happy with the setup, but would you mind @utchbe answering a few CI/CD questions? I mostly have text book answers to these, would like to here your take.

1. How is the release day experience of the release train? Is it stressful? What happens if a feature can’t be fixed in the 9 days? Is that common?
2. Have you considered decoupling the feature releases? Aka releasing features separately.
3. Cost wise, do you think it would make sense to make the 11 envs on-demand, even with the upfront investment?
4. What is the level of automation in your tests? How scary would it be if let’s say from tomorrow you do continuous deployment with the level of automation you have today?

Others, please feel free to chime in too.

[utchbe (Butch Mayhew)]

First off I would say for my context the way we are doing things makes the most sense. I’ll try to articulate the
’why’ based on our context but I am sure I won’t hit everything. For our biggest product, our infrastructure is hosted by a traditional hosting company (not the cloud) on semi managed servers. We have 4 web servers behind a load balancer, multiple DBs across multiple clusters, a utiltiy server, nightly job server, and an app server (endpoints/api). We try and replicate the same environment for QA so we are as close to production as possible.For QA environments we share servers for multiple environments, we will host 4-5 websites per web server. For some of our smaller (newer products) we are fully in the cloud AWS in docker containers and automatically spinning up test environments is a practice we have followed. The below answers will be in reference to our bigger enterprise product.

P.S. We have a lot of rules and process around our release train process but we also ok with breaking the rules knowing the rules are there to help mitigate risks.

How is the release day experience of the release train? Is it stressful? What happens if a feature can’t be fixed in the 9 days? Is that common?

Release day is quite calm. We have a code cutoff day 5 days prior to the release where we merge our Beta code branch (Hub) to staging, and have targeted regression testing prior to the release. If there is a bug we found in regression or after the code has already gone to Beta, we will revert the code out of Beta if it doesn’t need to go to production. This can get tricky and risky but it doesn’t happen often, and we have accepted the risk with our approach. Another way we help mitigate risk is if it’s a new feature we are working on, we will put the feature behind a feature flag which we can control either as an on off switch product wide or per customer. We like to do this to get feedback from our customers and this allows us a way to release the changes to a subset of customers. If it’s a new feature we are working on that is ‘dark’(behind the feature flag) and it has known bugs in many cases we are ok with releasing that code with the release train.

Have you considered decoupling the feature releases? Aka releasing features separately.

Sometimes this just happens where team 1 and team 2 start on different features at the same time but end up releasing at different release cycles. We have considered this, and have done it in the past, but because we serve a large customer base on an enterprise level, we have an extensive release communication process that our marketing team puts out with each new feature. That has been more important to actually communicate the value of the features that are released each release. We do have and use a ‘hotfix’ branch/environemnt for any bugs or issues that come up and may need to get coded/tested prior to the next release train. We will branch that off of Master if we know we will deploy before the next release train.

Cost wise, do you think it would make sense to make the 11 envs on-demand, even with the upfront investment?

Because we have shared resources and they are all located in our office on servers that are always running, we’ve already paid the high upfront cost for the servers. We do have an easy deploy process using a interal tool we built to utilize jenkins to build any code base to any of our environments easily. This is more important to me to have the flexibility to deploy any code branch to any environment. Having an environment always available (without me having to go click a build/spin up) button is big too. Our build/deploys take 15-30 min depending on what all needs to get built (legacy application built over the last 14 years). Spinning up new machines + building + deploying I would guess that would be around 40min.

What is the level of automation in your tests? How scary would it be if let’s say from tomorrow you do continuous deployment with the level of automation you have today?

Here is a talk on our automation framework and some of the things I’ve learned along the way -https://www.youtube.com/watch?v=-0nN0Twi6sU. I have pretty high confidence in our automation, nothing major would be broken. After initial testing of whatever changes, I would feel pretty confident in releasing right away.

[laszlocloud (Laszlo)]

Butch, big thanks for the thorough answer. Seems you guys have a solid process.

Big ups for feature flags. I’ve been geeking out on dynamic environments recently - the on-demand env per feature approach, but I think the next step will be to explore feature flags a bit more.

The reason I was asking is because I had hectic experiences in two separate companies with the release train. Both were unorganized places though: one was super young and was taking pride in heroic efforts, the other was just a classic place almost from Dilbert. Neither planned 5 days to iron out regressions after code cutoff. The startup had like a couple of hours, the other place like a day or so, but the release was always postponed… Both places would have been a nicer experience if the time was allocated for the release process imo.

In the startup after some time we decoupled the feature releases, and had dynamic environments to test each one separately. Which worked well, there were like six teams and they could deploy anytime they were ready. (feature flags were used to coordinate with the marketing releases, if it needed to have a proper launch, as you wrote it too). It did solve all the coordination problems, and things were not hectic anymore.

Testing though was not perfectly solved imo, there were unit tests, but the regression suite was not necessarily up to par with the release strategy. With the release train we could do the necessary manual testing, but when we did several deploys a day on the various feature branches, the effort needed to have the same level of manual testing multiplied too. I don’t think we did as much testing in that approach, given our regression suite was not that advanced…

It feels to me that you guys actually have more things in place to do continuous integration/deployment than we did Extensive use of feature flags, plus the confidence in the automation suite are two very strong signs imo.

Thanks again for the thorough answer!