How do you keep testing grounded when you don’t have realistic production data?

Discussions
,
1

In a lot of environments, especially where systems deal with sensitive information, regulated data, or government integrations, testers don’t get access to real production datasets. Instead we end up working with synthetic data, partial exports, or carefully sanitized samples.

On paper that sounds fine, but in practice it often means edge cases only appear once the system hits real usage. Strange formatting, unexpected blanks, weird encodings, inconsistent records, or volumes that nobody anticipated.

I’m curious how others handle this in their testing strategy.

Do you invest heavily in synthetic data generation?
Do you build libraries of edge cases over time?

2

I think that often testing does not need to be grounded. The nature of data is a big way that we interact with the product. Considering edge cases should be part of nearly any decent test strategy. Production data is fantastic, but if we use only production data we only find edge cases where the vulnerabilities just happen to match the threats produced by data we don’t actually control. Especially if the testing comprises of narrow capability checks. That’s only really acceptable if we don’t care too much about the state of the product.

Often it can be useful to use self-referential data to follow data flow and observe processing easier. Sometimes extreme data testing in a defocused fashion can find more problems quickly. Sometimes known risks are exercised with data given small changes. Or a combination of all of these during bug investigation or following risks.

Ideally I would want synthetic data generation. A smattering of databases in various states. Maybe a tool to put the database in a particular state. Perhaps a randomised data set for defocussing…

Well, ideally I want to be able to fully know the state of the product, be able to put it in any state I choose and process it any way I please. From that impossibility we work within our resources.

Libraries of cases are certainly part of my testing, as I want to build up an idea of what’s going wrong, what tends to fail, so that I can improve my testing. When a bug is found in production that’s something we can now anticipate next time - the process should absolutely be self-healing.

I’ll also say that randomisation, fuzzing, extreme data and behaviours, disregarding protocols, state abuse through multiple access points, hacking and other mayhem is a fantastic way to bring unanticipated problems to light. If the process is vaguely in place then it’s anti-fragile. The harder we hit it the stronger it will become. There can be an emotional factor where people want to treat the software delicately (especially if it’s tested by people who design/build it) - but users will not.

Beyond that there should be time in testing to try to anticipate them. That’s part of the built skill of testing, using an understanding of the product, company, environment and users (with some experience, empathy and some lists/mindmaps/notes) to consider alternatives to the accepted reality (and illusions) of the product. I use the HTSM and similar lists for this all the time - what have I not considered, and would that be important here?

Obviously nothing comes for free, and if we don’t invest the time and resources into testing properly then we have to accept that capability checking with an exploratory afterthought will consistently leak problems into production. But perhaps that’s okay too. It’s not always most profitable to be good.