In most areas of testing, the test environment should, in fact, remain consistent as regards its configuration, settings, and variables to obtain reliable, reproducible results during tests.
As I know, in a control test setup (much like in science), we consider the test environment as a constant, the tester as the independent variable, and the test result as the dependent variable.
The question I have now might sound obvious or maybe silly but I hope you’ll indulge me: Are there valid exceptions to this rule of consistency?
If, say, I am testing an app where some settings just don’t seem germane to the feature under test (at least from my angle), may I alter them? Or should I always consider every setting as critical to remain constant?
I’m sure everyone has great stories: especially those instances when an inconsistent environment may have actually been useful in discovering some information.
Thanks ever so much for humoring one such curiosity!
Even for that is no guarantee. It’s constant to some degree and there are factors out of your hands.
E.g. network delay, actual CPU cores or RAM available for a VM, OS patches etc.
Every test environment is dynamical to some degree.
In my experience testers have less control over the test environment than a scientists.
Why should you limit your testing and not look for risks?
Act with care. Inform all relevant people and revert the changes onces you are done.
Treat exceptions exceptionally.
A big problem testers do face, is just sheer amount of system and environment knowledge needed to make good and educated assertions around desirable environments and behaviour. I mean it’s obvious if some resource randomly is constrained, it’s going to make tests flakey, triage harder, and slow down some testing. But you will end up learning a lot. On the other hand making your test resilient to resource problems by defensive coding may end up hiding bugs. We sometimes do see retries in test code, some frameworks even have a retry built in, which tells me they are not testing the user experience at all.
I prefer to do all my exploring in a more random environment, anything that might just let me experience what customers who did not splash out on 10Gbit ethernet, is likely to experience.
You could look at this through the lens of focus/defocus.
Having consistency (in some subset of the test environment over which we have agency) is a kind of focusing. We change one factor at a time, simplifying our testing, conserving states in order to make precise observations. This gives us test integrity. The knowledge that our inferences, based on our observations, are likely correct. A logical line from one to the other. Also if we alter environmental factors one at a time and make observations that’s a focusing technique, a kind of progressive mismatch where we make single changes and observe the results, so that we can be more confident that the change we made is both known and responsible for the effect by itself.
Defocusing would be changing multiple factors, violating patterns, broadening observations, changing lab procedures. It’s about complex actions. The goal is to make the test harder to pass, to find unexpected or elusive problems and find more problems quickly. If we change environmental factors a lot all at once we might discover ways in which environment seems to have some impact. And if that impact is important we can use focusing techniques to figure out what’s causing it. This also has ties to the phrase “it works on my machine”, changing that environment to another machine is a defocusing technique that helps find differences in environmental effects and a starting point for investigation.
There’s no reason why we cannot change up our environments (in possible ways) if we want to see the impact it has on the product, when we choose to.