My company has recently built the latest release of our software and it’s been put in front of users during a training session and then they find quite a number of really obscure bugs. This is after the usual suite of automation & exploratory testing has been run on it. As much as I try of think of ways of how I could have found them myself I think the solution really comes down to we need real users testing our software earlier in our processes. I’d love to hear some thoughts and ideas on how we can get users involved earlier in the development process before those bugs reach production.
2 Likes
It’s a really interesting one - I would have a look at the types of bugs they are raising. Was it missing in the requirements? Is it related to usability? languages? etc Having a lessons learned on each bug might be helpful.
Do you ever get people across your company to test the product? It could help getting different perspectives in on it.
Could you do a User Acceptance Testing (UAT) phase, where you invite your super users to come and test the product on a pre production environment? It might help with customer engagement too, so you could float that idea about.
3 Likes
I will suggest look for bugs, are they really bug or a type of enhancement
also go for A/B testing where you can rollout feature to some specific users to get real-time feedback.
Also try to do RTM (Requirement traceability matrix) so that as a QA you know that your are testing the correct
1 Like
All the suggestions are great. Wanted to mention, does your organization have alpha/beta version/testing for the software/product? That’s another option, to have a product alpha (testing) program, where interested users can sign up, and provide feedback and report bugs. You can even offer incentives to recruit such users, such as free use of product during these stages, or pay crowdsourced companies to get you such users to help test during these stages.
Beta stage the product is more ready for general users to use and test against, and is there to catch bugs missed during QA testing as well as seek improvements for product or correct any incorrect implementation. Alpha stage, the product is expected to be potentially buggy and not all features complete.
We try to learn from bugs found by users. Some are things we missed and should pay more attention to in the future. Most common is keyboard navigation vs. mouse.
We also have a beta test phase where customers can install our new version. They have 6 weeks to give us feedback. Sometimes they are interested in the new version and sometimes they need to be “bribed” at bit (free setup support, choosing bugs to fix).
I was going to suggest this too (David Luu’s suggestion). I first heard about this at a conference where Blizzard was explaining how they use this internal testing strategy.
Aside from having real users test the system and provide feedback. It might also be good to consider the following:
- perform user studies with the users to learn how they use the product, incorporate those ideas/cases into your internal test strategy
- do post mortem / assessment on the user reported feedback and bug reports and incorporate those cases/findings into your internal test strategy. As part of the assessment, find out how this was missed during internal testing
- try thinking like more like your users as part of the internal testing phases, if possible/applicable
- as part of exploratory testing, look into possible ways to use the product in ways possibly not accounted for.
It could be your internal testing strategy doesn’t test enough user behavior cases or misses some interesting edge/rare cases.
You might be surprised to find how users might use the system in ways you don’t anticipate, or how impatient users could be in using the product.
Another area to be on lookout for is when doing product integration between components or say an integration of 2 products into a new version of some product. These are notorious for overlooking potential (new or unaccounted for) usage scenarios now enabled through the integrations. The devs just integrate the code, business unit and project managers just on high level decide they want to integrate tool products and spec out the big picture but don’t cover the fine little details. Once you integrate the 2 products, sometimes it results in new ways to do things not possible prior to integration - you have to catch that in QA by thoroughly considering the workflows before this gets caught by customers who might come across the new scenarios. I’ve seen it happen at work before.
If your product implementation supports it, you can also build user behavior tracking, with the personal identifying information (PII) removed (or not included). Basically tracking how an anonymous user goes through the flow of your product (which paths are taken, what is clicked, etc.). This is also another way to find out how your users are using your product, anonymously, so you can account for that in the internal testing.
A very interesting topic, and one of my favorites too.
In my previous companies, we used to have something called “user acceptance testing”, which is not necessarily a training, but an exploration session for the real users using the software, and the development & testing team observing their usage and making notes. But as you pointed out, it would be too late to find those bugs at that point and it would be great if users can give their insights about their usage and understanding of the product well in advance.
Some tips:
-
Requirements analysis, prototyping stages have the opportunities to engage the users.
-
Continuous Delivery puts the developed product in front of the users as soon as possible so that they can give their feedback.
-
Ensemble (Mob) programming sessions can include users (subject to confidentiality clauses signed between your company and the clients) so that users can experience first hand how the product shapes up and provide their feedback right then.
-
If yours is a B2B product, it’s easy to bring in select users for them to experience the product at various stages of development. If it’s a B2C product, significant amount of work is needed in locating the target audience who can give feedback.
Hope that helps. Feel free to get in touch with me at https://venkatramakrishnan.com for detailed inputs. Happy to help.