What are things need to be consider when we are setting up the QA environment
- Compute resources
- Third party software versions
Etc
1 Like
What is your context or motivation here @kayu ?
Test Environments are just that, a sandbox that is intentionally designed to meet business goals, if it’s a business goal to publish multiplayer games, your test environment would want to replicate your target audience. If your business goal is to provide a web-service for weather apps, then your test environment will be designed to facilitate testing of the web service infrastructure parts that matter to the business. If this means 3rd party components, then make sure your environment lets you control all 3rd party components to allow swapping out experiments. A thing to bear in mind when specifying test environments is to note that there is never any “single” environment, so any environment spec needs to make things as cheap as possible, or else you run the risk of the environment becoming unavailable to it’s users because your team cannot scale. This means custom environments that target certain kinds of experiments around things like resource usage.
So not only be prepared for many environments, but also be prepared for variations, but under it all, design your environment to be dead easy to monitor and to modify. As well as fast to build out. Which will often mean test environments have varying security and business rules relaxed so that experiments can be more rapidly executed than would in production.
4 Likes
As a Tester we have to set up environment to deploy the release and perform our testing. Is that environment should similar to production ?
1 Like
I think we are talking about personal environments a bit interchangeably or confusing them with things like environments that are part of your CI/CD process like the “integration” environment and a “staging” environment. As a tester, your personal environment needs to be cheap and lightweight as possible and only needs to change over time to better catch any bugs that actually escaped.
The real world is often context specific and your context matters. Environments in many teams are often hybrid, where you don’t end up controlling everything, but end up sharing some parts. Sharing an environment can also be an advantage, especially if it makes you think about testing at scale more carefully if you have to share a resource. Private environments have their own disadvantages, I’m a fan of mixing it up.
I think to answer the question, “Should it be like production?” , that “Depends” is not a good answer. Mainly because it’s that trade-off of financial cost to copy production and how many more bugs that will actually help you to catch. It can lead to bikeshedding and avoiding the issue, and distract people from time better spent hunting bugs if your environment is very hairy. Generally it’s possible to catch loads of bugs in a much smaller and vastly-scaled-down environment, than in a realistic one.
2 Likes
As @conrad.connected says, it’s a trade-off. The factors can involve data security, privacy, systems access, permissions, server costs, tech stack complexities, automation hooks, realistic user data vs test coverage, shared configurations, controllability issues, controlling performance bottlenecks. You look at the needs of everyone, development including testers, operations, management, IT, whoever, and at the costs, and you try to iterate towards the best solution.
I think that production-similar environments have their place, depending on the testing that you’re doing. Depending on your context there are a somewhere between a zero and ton of problems that only a production-similar environment will find. Also depends on what “similar to production” means. That could be on the same kind of server, or using user data, or carrying a similar amount of traffic, or running for a similar length of time, or interactions that users typically perform, or settings/configs that users use - any of those, and more, in different combinations and configurations. It depends on what you want to achieve in your testing.
Sometimes it’s covered by the test coverage itself. It may be that you want to repeat some testing with different settings or configurations or toggles. You could do this with an environment of user configurations to find problems, or if you have the capacity you could test with a set of configurations you feel provides coverage which are informed by your understanding of user configurations.
Perhaps I need a system running on a copy of a production server because I’m testing the breaking point of that server - how many concurrent users before the server can’t cope, and what happens when it does? If it automatically expands and contracts to serve user capacity, does that impact users or data? For that I need a system that’s similar to production in some particular ways because it’s a real-world reliability test.
For very focused testing of specific functions in specific configurations I want my own, specified, self-referential data. For defocused exploration I might want user data, to find more problems faster. Or a combination.
So every step away from production is a real difference away from the reality of the use of your product, but not always in a way that matters. You need to decide that difference for yourself, and then if the difference is important enough to warrant the cost of accommodating it.
3 Likes
If you want to test the functionality of a service, having an autoscaling cluster with 30 nodes is almost always a waste of money. If you are testing the autoscaling of the cluster, having the correct payment integration service might also be a waste of money. If you go for, “I want a production like environment”, my experience is saying that you will end up with an overpriced environment that still cannot test all you want.
It this is about a personal test environment I favour the ability for you to quickly setup an environment and having the control of what components you have faked and what you have “real”. But if that is not an option the most important aspect is that you as a tester understand what is the limitations of your test environment and what is the difference between your test environment and a production environment.
3 Likes
I agree on this totally concerns like
- if your system connecting 3rd party api & using the simulator for that; how will it impact your testing ?
- if you are having the multiple sites in production and your micro services want to share the details between the sites ; if you are not set up as production how is this can testable ?
etc …
1 Like
Is that mean understanding the limitation on our testing ? due to this there can be issue raised in production .
Given the choice, I’d prefer my main test environment to be a much lower spec than production - if test works fine with the anticipted number of concurrent users volumne of data anticipated in live then live should be more than capable.
I also prefer having a multitude of test environments available (every combination of server/database version that we support the software on if server/database are a feature) and have automated software builds deploying to each overnight to ensure that we’re not writing code that’s version specific.
It’s not just which test environments though, it’s how much access we have to those environments. Usually it’s full administrator access so that testers can monitor the entire system while testing software but we have one environment where we don’t have that level of access and it scares the heck out of me because I’ve absolutely no idea if we’re close to the operating capacity or not. Next aim is to get that environment fully open to the QA team.
3 Likes
Chris has pretty much added more detail to the question about how accurate a copy the test environment should be in terms of security, complexity, performance . Basically speaking testers will remove complexity from their personal environments, this makes loads of sense to do. For example, you often want to disable security barriers, so that test workflows don’t have to go through security hoops to register test accounts. Every time you remove complexity from your environment and from your test setting up steps, you speed up your test execution. It’s a trade off.
I’ve worked in places where the testing environment is so simplified that it has little resemblance to the live environment. Which is great for running experiments and finding bugs faster, getting logs out faster and so much more. BUT, all this points to is that testers need a really good understanding of the product architecture and a good understanding of the hacks they make in the test environment. As Chris explained, it’s a trade-off, and often means that for example that different test teams will have different environments. Basically don’t waste time testing something that your team does not directly integrate with, so remove those from your environment. But know very damn well what things you disabled.
1 Like
Agree on this . We had concern some bugs raising on prod environment how can we reduce that ?
Example
In QA environment if we are using simulators for third party but production it connect to actual service , If say third party sending the unexpected error but that didn’t written in our simulator.
if it not properly handle in our system , this bug will raise in production.
1 Like
yeah - I hate simulators, but I also love them. Any tool that helps you find bugs sooner is a win, but when raising a bug ticket, always have to be sure to verify the bug also reproduces in the real world. Knowing the limitations of the tool is king, choose the right tool seems to mean environments too. I had never thought of tools that way actually.