Greetings everyone,
First, a confession. Iām not a tester; Iām a developer ā¦ (Impostor!).
Imagine a piece of creative software, something like Photoshop for the web browser, where end users have a lot of freedom. This allows for highly variable workflows that, on the surface, seem nearly impossible to account for with pre-planned manual regression testing. The cost of sitting down and mapping every permutation is too great. So when something comes up and weāve got concrete repro steps, we add a test case and make sure it doesnāt break in future releases. But what do you do when you just canāt make it happen?
Sometimes, despite our best efforts, we canāt get to the root of an issue. Users are in different time zones and geographical locations so we canāt easily look over their shoulders when they donāt know how to make a bug happen, but they know what it looks like when it does happen. In tough cases weāve stayed up late, recording screen sharing sessions, essentially watching someone work for hours, hoping the thing will break. In rare cases weāve flown testers or developers to do this in person. Using these techniques we end up figuring it out, but Iād like to know the experts (you!) handle situations like this. I imagine there are cheaper and faster approaches that can be used.
Iāve done research online of course. Here is a sample some of the advice I found:
- How to handle unreproducible bugs
- What do you do when you canāt reproduce a problem?
- What Do You Do When You Canāt Reproduce a Bug?
- Fixing Bugs - If You Canāt Reproduce a Bug, You Canāt Fix It
My takeaways are:
- Record video
- Add logging
- Use fuzzing
Weāve tackled this from the dev end, adding client and server side logging, even a client side exception global handler that makes it easy to copy/paste unhandled exceptions, but a lot of times, the best we have is an artefact in a broken state and we canāt figure out how it got there. Weāve asked our QA team to jump in with exploratory testing and they are a great help in most cases. Sometimes though they canāt figure it out either. I donāt like the idea of an end user recording their own video for hours; Editing and shipping to us for analysis as it seems like too much of a burden on their end. Iām unfamiliar with fuzzing, but that seems to deal more with form field inputs if I understand correctly. In our case users are manipulating layers and graphical objects with the mouse a lot, so it doesnāt seem like a suitable approach. What would you do in this situation?