For a client we have build a “beta testers program” where users could sign up to access new features before they became mainstream available. They also were offered an option to be tracked during their visit on the web application as a way to improve the service and discover flaws.
With only 8% of all users signing up for the program, we had detailed end-to-end access flows through the whole web application giving us repeatable usage data that we could use in our automated test tools (like Selenium) and issues that these users generated (404 and 500 errors) were added to the priority issue tracker, with a full detail about which pages they visited, what button or image they clicked on and which components were affected.
We discovered many small but very impacting issues that we would normally not find in our pre-production tests. We also learned that users don’t use an application the way developers, testers, business owners or project managers think they will. This type of insight in user behaviour on the production web application thought us many things and made the overal application better for the end-user.
In this thread mentions of Netflix Chaos Monkey were made as well. We consider that to be part of our resilience tests to find flaws in our design and dependencies when parts of our application architecture are no longer or only partial available. It’s a necessary step to ensure that we fail safely or that we have mechanisms in place for when a dependency is not available. This has thought us to implement heartbeats and deferred execution solutions in our application architecture.
One important piece of advice: when you want to implement such a “beta program”, especially with the upcoming GDPR and other privacy regulations, make sure you have an explicit approval from the user as you will gather a lot of information, often sensitive PII. And have people re-confirm their interest every 6 months or year as most people forgot they signed up for the program.