Sandbox testing - Totally unstructured?

(john ) #1

Well am just entering into another phase of sandbox testing (in a project where I am just doing holiday cover). The current test manager, believes that hand over the sandbox and let them have fun with it, is the way to go. I kind of like to have some loose structure around it, so as to give me a degree of comfort as to what is being covered.

What are the general views on sandbox ?

(Chris) #2

Is “sandbox testing” a thing? If it is, could you describe it? My definition of a sandbox is an environment isolated from production and development. This permits you to do things that are not possible or desirable on those environments. For example if you want to spider a website you don’t do it on production, as it might open the admin page and click all the delete links on all the users. So to me “sandbox testing” is any testing done on an isolated environment because it can’t be done elsewhere.

So first I’d need to know who’s getting this isolated environment, why it was built, and what you hope to find. That’ll guide how you go about things like coverage, observation and reporting.

(john ) #3

The situation I find myself inheriting is this.

The Supplier is currently working on Sprints to deliver new functionality. In parallel to this, they are giving us a new code version release, to bring our current base code up to date.
The thinking is that to save time in the future, this new versioned code, will be dropped into the sandbox and the business ‘have a look at it’. Currently there is no appetite to allow them to raise defects out of it. Which I support, due to data maturity, interfaces etc.
I can understand the desire to get an early look at it and to feed back to the suppliers. However, like yourself I am not too sure the two sit together well. Either it has some structure to it, or it is basically a training / knowledge enhancer for the new version of code and functionality.
Reading the test plan / approach, it basically looks like the desire is to shorten future regression by letting them loose on the sandbox as early as possible and getting feedback from that. Agree with your thoughts, hence my predicament. It’s either ago and play phase or it is a testing phase. If it is going to be something between the two, then some structure is needed (?)

(Chris) #4

So they want to run lean by getting early feedback, but they don’t want that feedback in the form of bug reports. I find that very confusing.

I’m entirely in agreement with you. Without any process or structure it’s a waste of time and energy. If they want feedback at all then the minimum that’s needed is some way to record the problems that are found. If they want feedback on all of the functionality then some way to ensure users have explored all of the areas is also needed. If they want directed feedback about a particular area or quality criterion then direction of the users on the sandbox is also needed. You could sprinkle in screen recording, note taking, exploratory charters, product area checklists, risk-based testing… It’s really a case of purpose. Someone needs to shout “why?” at someone else and then hopefully the fog will clear.

Consider what’s best for you, your end users of the software, and whoever signs your paycheck. That’s a useful starting point. If it would serve you better to give fuller feedback than a checklist of tasks (scenarios the users will have to perform on this new functionality) might be helpful. If it would serve you better to give more accurate feedback then a charter or questionnaire based on quality critera might help (“How easy was it to X? Did it install okay? Was it fast enough for you?”). And so on. The HTSM is a really useful guide for this sort of outside-in risk analysis (

(Alex) #5

I agree with @kinofrost! I think you need to ask more question such as the ones he recommended. Who knows it may be that they have already thought this through and you can further understand their purpose and thereby you can provide better feedback. Or if they have not thought this through, through your questioning you can help them realize that their idea is flawed and you can possibly provide an alternative that will be better suited to their needs. Keep us posted!

(john ) #6

Thanks Alex - time will tell !!! (well time and fortitude)