Exploratory Testing Tools

I’m keen on making my exploratory testing expeditions a bit easier… From gathering evidence to reporting it to my test case manager, I find myself using a mishmash of tools and still doing a lot of things (like uploading evidence) manually. I’m wondering if other people have these same issues?

Currently, I’m hacking together a tool that helps me capture and collate my evidence and details about my test session, then export those to my TCM.

But I’m really curious to hear what other people are doing to make their exploratory testing sessions seamless??? What works well and what do you think the current tooling out there is missing?

3 Likes

I took a similar approach to yourself for tooling. I’ve created my own tool that lets me write different notes and insert images (i.e. screenshots) and eventually save as a PDF. My team have responded well to my reports with the tool.

I’ve looked at more professional yet free tooling but (unsurprisingly) haven’t found anything that seems as easy to use. I believe there’s some great plugins for browsers though, however for myself I don’t actually use a browser as I’m testing services or old school desktop applications.

I’d also be really interested to learn what tools people are using.

2 Likes

1: Reduce non-testing activities
I’d say that every second you spend writing notes, capturing evidence, writing bug reports and so on is time taken away from your testing. Your easiest saving might be in taking fewer notes, for example. I’d look at what’s mandated and why - process improvement might mean that the savings on fewer notes outweigh the benefits of taking them.

2: Reduce limitations on non-testing activities
The less prescriptive a system is the better. Testers should do whatever works for them as much as possible. Restrictions on file formats and technologies mean that a tester may have to work with a system they don’t like or doesn’t fit their methods. Forcing certain tools on craftspeople is bad and you should do it no more than is necessary.

3: Determine what you can change
Now you know what you need to capture and what restrictions you need to work with - but you might be able to change those restrictions if you have a better idea. For example if you’re told to upload all the files one by one to an internal website form and you need to upload notes, documents, a video, screenshots, bug report links and whatnot it would be easier to copy them all to a shared drive under an auto-generated folder name and software can do the rest - but that involves not using the form you’re told to use so you’ll have to go to the manager with that, and it involves having access to a shared drive across the test team (plus anyone else who needs access to those files) which would involve working with the IT & ops sides of things.

4: Try new things
When you’re settled down between what you changed and what you can’t change then it’s time to look at ways to improve things just for yourself.

The way I used to do things is take a template I made in OneNote, make that into a new page in OneNote, take all my notes there, paste any screenshots there, turn the whole thing into a PDF and attach it to the story in JIRA. I found LICEcap through another tester, so asking around is a good way to find tools you might use. Some tools depend on what I’m doing, sometimes screen capturing is a useful thing to do. Sometimes I had tools that could dump information to files to expose the internals of the software, or working with things like wireshark that have exports. I think it’s a good thing to have a mishmash of tools for this sort of thing - small tools that do very little but very well that are suitable for purpose and low-friction are important. It’s tempting to want to pool them together but I find anything that says it does a lot usually does at least some of those things in a way I find annoying if not unusable. Because I like lots of niche and custom tools most of them are unsuitable for anyone not doing the same thing I’m doing.

Things I like that I’d suggest to someone else
Some of these might be out of date by now, but you can use the ideas to find a replacement

OneNote - my favourite note taking tool
Notepad++ - find in files feature is nice
LICEcap - low friction screen capture tool for taking animated GIFs of parts of the screen. Great to communicate how to recreate a problem, or provide believable evidence of a problem.
WinMerge - for highlighting differences in files, useful to compare logs or see what changes happened in other files.
FRhedit - same thing but for hex values
Excel/Google Sheets - data processing and visualisation
Xmind - my mindmapping tool of choice
Baretail/Kiwi - a nice log tailing tool with customisable colours, letting you highlight errors in a log on the fly or better show message types
InCtrl5 - This is probably out of date now, but it would take snapshots of folders and the windows registry and show you any differences. Useful to track system changes.
slipsum.com - a site to generate lorem ipsum text except that is uses profanity-laden quotes from Samuel L Jackson.
Spy++ - This used to come with Visual Studio and was good for watching windows messages

Most of these tools have some capture system or export system to get this information more easily into your notes and make it simpler to report on them.

Other advice
There are a bunch of full screen recorders out there that could replace the need for taking notes on what you actually did with a video of it. Although that won’t capture why you’re doing it it will mean that you can just timestamp your notes and then have a file showing exactly what you did.

You can automate some of the problems away. For example if you have a way to scrape your notes for timestamps you can autofill metadata about your session - just be careful that automating away your attention doesn’t lead to more problems.

Templates are a nice way to reduce the amount of stuff you have to write every time, and act as a reminder.

Checklists are a nice way to remember what you have to do after a session. I had one in a template that included things like “get this session reviewed”, “Report all problems” and “upload PDF to story” which I’d have to check off to finish the session. It’d also have a section called “Environment” where I’d write the area of the system I was working in and what version I was looking at.

3 Likes

I’ve been looking for a good free tool as well, and haven’t found one that covers all of the use cases (or most of them) yet, so I think I’m going to put mine out there and see if folks find it useful enough to give me feedback so I can continue to make it more useful. :slight_smile:

I’d love to see what your reports look like and what data you’re including! Could you share?

Wow - this was really helpful, thank you!

I love the ideas around automating as much as possible and the QoL features like templates and checklists!

My thought was to put everything on a timeline to preserve as much context as possible. Screenshots, screen recording, audio notes, and written notes would all live on the timeline.

My initial instinct here was to connect this with all of the major test case managers because everywhere I’ve worked has used one or the other, but the point is well taken that some folks just manage that all internally… What other methods of automating the data upload would you suggest supporting? (Write to filesystem? Configurable POST to API? Something else?)

I’m also very curious - how do you like to report bugs in a way that minimizes time while testing? I could imagine filing a bug report via drop down on a specific piece of evidence, but maybe there is an easier or more appropriate way?

I can’t share an actual report as they tend to contain internal only information but I’ve put together a quick report of my tool, by my tool.
ExampleDetectiveVisionReport.pdf (241.0 KB)

Edit: I need a new icon. That is the character from a dinky game that I made called “Fergie the Gum Ninja”.

2 Likes

Hi @dacoaster,

A while back myself and an ex-colleague of mine created a tool called TestBuddy. It aimed to help folks plan a little and simplify how they captured their long-form testing notes. It also endeavoured to enhance debriefing efforts.

While we had some paying customers, we ran out of cash and energy and shelved the project.

It’s no longer available yet this YouTube video might give you some insight into how I (at least) capture notes during an exploratory testing session. The closest tool I’d use now would be to just crank open a Google Doc and use the same section titles [Planning, Exploring, Summary] and PQIP (Problem, Question, Idea and Praise) labelling. Then export to PDF.

This is great - thank you so much for sharing!

I like the ideas around choosing PQIP for notes and ensuring it supports good rich text editing.

I’m hopeful that I’ll have a first draft ready to share in the next few weeks. I’ll report back here when I have something to show.

What are your thoughts on the best way to share the reports? I was going to give the option to automate uploads to test case managers/JIRA, but how do your teams usually collaborate around things like these reports?

2 Likes

Thank you for this Simon!

I love the TestBuddy concept.

In your view, what was the “biggest win” the tool gave people? What kept the customers coming back to the tool?

2 Likes

Thanks! :blush:

Our customers at the time loved the simplicity of it. As in, they found other tools full of too many features. They also enjoyed how easy it was to export to PDF and share with others. It seems it helped start conversations amongst their engineering teams. One customer told us their developers would always ask for a TestBuddy report. That was a lovely feeling.

I’d also recommend having a chat with some of those old customers if that is useful. For privacy, I won’t share who they were yet perhaps putting something out on various platforms might help.

What other methods of automating the data upload would you suggest supporting? (Write to filesystem? Configurable POST to API? Something else?)

Not many people will say this but I don’t think it really matters too much. It’s more likely to be what you know and have, plus whatever everyone else is able to run. I like Ruby so I wrote a lot of tools for myself in Ruby, but that meant that nobody else could run them without having Ruby. Web is great for that if you have a machine that can run the software that can remain on all the time, even when you’re not around, and has multiple-people access for maintenance and so on. But again I think that doing it is more important than perfecting it because it has a non-zero chance of not catching on and you need to try something else that works for your particular context. It should definitely be easy to use and mandated from above because people are very good at ignoring suggestions. I find that lightweight, modular, adaptable, messy systems are less cool but work more often because they morph to the needs and wants of the people that use it. Monolith complex systems, even ones that look simple, I find to be the worst for this because they rarely survive contact with reality.

More important is actually doing it and getting the buy-in from someone whose job it is to manage that sort of thing.

how do you like to report bugs in a way that minimizes time while testing?

By not writing a report. It’s much faster just to go to someone working on the system and tell them what’s going on and have them fix the problem, or request more information in which case you’ll have to write a sort of report anyway. After that it depends on what the problem is - I find that understanding how bugs work helps to communicate them simply. Understanding how other people want to be communicated with is very important for writing suitable reports. Also low-friction tools as mentioned - I find an animated gif thrown together in LICEcap is much easier to both report and understand than a list of text.

It’s also important that the communication comes first. No tool or system should get in the way of communicating important information, which is what bug reports are for. Computer programs are very prescriptive and slow to change which is the opposite of how people work together.

I just upload them onto the story rather than anything automated. I also keep them in my usual docs storage as it is easier to look there than remember which story I attached them to 3 months ago.

I don’t use the tool all that often, only if I’m picking up something beefy, and I’ll usually pull in a couple of interested devs to talk through the findings. No one else uses the tool so far. I’ve more work to do on sharing exploratory testing with the devs and there’s no other testers that I actually know within the organisation (all part of a different sub-org in a different continent!).

I use Notion for basically everything at this point. I have a “Journal” setup for testing with some basic templates set up. The primary use ones are my testing notes and defect report. Things are in a single place, in chronological order, tagged, sortable and searchable, with templating. Being able to tie it into other tools, like github and jira, makes organization easy.

Thanks for the idea Simon.

I’ll ask around and see if I can find some folks who were using the tool to get some more feedback there.

I’d like to create something very similar in spirit to TestBuddy and open source it.

1 Like

Definitely words to live by!

I feel like many products fall into the trap of trying to be all things to all people and lose sight of user experience and ease of use for the average user.

1 Like

That’s very interesting that you decide to reach for other tools when the job is smaller.

Why do you think that is? Is it just too much cognitive overhead to fire it up? Or does it slow down your flow while testing?

I had never heard of Notion but it looks really interesting.

Is that a tool that your org uses or just something you set up personally?

Something I use personally. We were using it within the business, but moved away from it as it wasn’t seeing great use.

A lot of the user stories that I test are fairly well defined and there isnt a huge amount to test per story so I end up just writing my Jira comment as I go, possibly using Google Sheets if theres a bunch of permutations to use.

The times that I most value having a tool is when I’m picking up something that I’m less familiar with or there’s been more significant changes.

What’s the killer feature for you? Just that it’s all in one place? Or anything more specific?