First, a confession. I’m not a tester; I’m a developer … (Impostor!).
Imagine a piece of creative software, something like Photoshop for the web browser, where end users have a lot of freedom. This allows for highly variable workflows that, on the surface, seem nearly impossible to account for with pre-planned manual regression testing. The cost of sitting down and mapping every permutation is too great. So when something comes up and we’ve got concrete repro steps, we add a test case and make sure it doesn’t break in future releases. But what do you do when you just can’t make it happen?
Sometimes, despite our best efforts, we can’t get to the root of an issue. Users are in different time zones and geographical locations so we can’t easily look over their shoulders when they don’t know how to make a bug happen, but they know what it looks like when it does happen. In tough cases we’ve stayed up late, recording screen sharing sessions, essentially watching someone work for hours, hoping the thing will break. In rare cases we’ve flown testers or developers to do this in person. Using these techniques we end up figuring it out, but I’d like to know the experts (you!) handle situations like this. I imagine there are cheaper and faster approaches that can be used.
I’ve done research online of course. Here is a sample some of the advice I found:
- How to handle unreproducible bugs
- What do you do when you can’t reproduce a problem?
- What Do You Do When You Can’t Reproduce a Bug?
- Fixing Bugs - If You Can’t Reproduce a Bug, You Can’t Fix It
My takeaways are:
- Record video
- Add logging
- Use fuzzing
We’ve tackled this from the dev end, adding client and server side logging, even a client side exception global handler that makes it easy to copy/paste unhandled exceptions, but a lot of times, the best we have is an artefact in a broken state and we can’t figure out how it got there. We’ve asked our QA team to jump in with exploratory testing and they are a great help in most cases. Sometimes though they can’t figure it out either. I don’t like the idea of an end user recording their own video for hours; Editing and shipping to us for analysis as it seems like too much of a burden on their end. I’m unfamiliar with fuzzing, but that seems to deal more with form field inputs if I understand correctly. In our case users are manipulating layers and graphical objects with the mouse a lot, so it doesn’t seem like a suitable approach. What would you do in this situation?