I’m now looking at how we can help people who have to setup environments for automation. I want to help them understand the different requirements they need to consider when speccing out a test env. So I wanted to ask:
What do you need from a test environment to run automation against?
As always, I really appreciate everyone who reads and replies to these questions. The information shared is helping future automators.
I can offer a partial list, based on my experience:
Isolated - the test environment needs to be, as much as possible, only used for test automation. In the case of multi-tenanted cloud-based apps, one or more dedicated test clients that nobody outside the test team can access is essential to make sure the environment isn’t changed on you.
Dedicated runner(s) - the systems running the test harness also need to be isolated. In this case the isolation can be provided via the CI/CD harness or it can be physical. One place where I worked had a habit of grabbing every retired desktop system and repurposing them as test environments for their automation runs.
Enough resources - What constitutes “enough” varies according to the application in test. For desktop applications this can be quite large, where less is needed for web-based software. Software that’s got a lot of legacy code and very little that can be tested by unit tests or API tests is likely to require a virtual system that has registry tweaks to prevent screensavers or lock screens activating and to automatically log on the designated automation user - and may not be able to be run in headless mode.
Shared reporting location - test results need to be dropped in a known, shared location. If the CI/CD system and automation harness have reporting that works for the team built in, that’s fantastic. If not, it may be necessary to modify what is available or in extreme cases build your own reporting from whatever is generated by your tooling.
Control - the test automation team needs to have full control of the environment
Have to say, the number of times we run out of storage in our test environments is embarrassing, unfortunately more space actually postpones the problem, which really revolves around cleanliness. Today we are looking to use Ansible to attack the problem, the problem is that very few tools for host management are very portable-friendly. So for me, having a management agent and report tool is lately a new requirement I have.
Ideally, I would want two environments: one to develop my automation in and one to run it in… Everybody has a testing environment. Some people are lucky enough to have a totally separate environment to run production in.
In addition to the points mentioned earlier, I would emphasize that a crucial criterion for a testing environment is to ensure that only QA members are using it
