Masterclass Further Discussion: Explore the Unknown with Exploratory Testing

Tonight, @darktelecom will be joining us for a masterclass exploring exploratory testing :explorer: (can you say that 10 times fast? :stuck_out_tongue: )

If we don’t get to your questions tonight we’ll add them below. If you’d like to continue the conversation from the webinar, here is an excellent place to do that :wink:

Of course, the recording of the masterclass will be available to everyone in the Masterclass section of our website after.

1 Like

Looking forward to it :slight_smile:

1 Like

Wow what a session!

Questions we didn’t get to

  1. Typically, Test Management Tools focus on “pass or fail” for test cases? How would you suggest to handle exploratory testing in such tools, where there is no longer this clear “pass or fail”?
    1.1 Comment: Pass will hide the issues found. Fail will often “require” to make it green in my experience.
    1.2 Comment: In our pass/fail tool, pass means it has been explored and fail is the area couldn’t be explored due to a blocker. This does mean most things “pass”, however.
  2. Could you describe how and with who you do the debriefing?
  3. Do you have an idea how to help testers coming from a more traditional background to get started with Exploratory Testing and “defend it” against “management” that might be focussed on pass/fail statistics?
    3.1 Comment: Invite them over to a pairing/ensemble session :slight_smile:
    3.2 Comment: I would also add to Thomas’s question if you have some quick wins tips in order to “sell” Exploratory Testing more easily
  4. What in your experience controls the kick-off of exploratory testing? Like is that driven off a system having new features or maybe done according to a regular schedule like once a quarter or some other period?
  5. What is your opinion on doing exploratory testing in a pairing situation with developers? Is their knowledge of the software a curse?
  6. Xray seems a useful tool! Thanks! Does it have the ability to save exploratory sessions for later playback and sharing across the team please? If so, is it just locally, to a file share or in JIRA?
  7. If the system contains several websites and windows services - is it possible to record the test across these parts of the system in the same test run?
  8. How long would you do an exploratory session for? Trying to timebox them makes sense in my experience as this kind of testing is demanding so keeping sessions reasonably short keeps you fresh. Would you agree please?
2 Likes

Well, as I’ve shown, quality assessment is different with exploratory testing; it’s a true assessment, so it means we got our own perception of quality based on what we explored.
With a traditional test management tool, you could for example create custom statuses to indicate an higher or lower confidence.
You can still mark it as FAIL if you have a blocking problem there.
Marking it as PASS, as you’ve mentioned, can hide issues. But you can also create a PASS_NEEDS_DISCUSSION for example. I would advise you to try what works with your team. It also depends on how you share/debrief afterwards… you may want to have a more visible status to signal something or not.

1 Like

Well, whenever debriefing it’s important to have there the people that were mostly involved in that feature/story/etc.
So, having the developers and testers makes all sense. In one of our teams, we do mob-programming with testers also involved. So feedback comes early. But later on, whenever exploring it in more depth, maybe we need to have the product owner involved. I all depends on the risks that were addressed. If some usability issue comes, maybe we need to bring one of the experts to the discussion.

1 Like

In terms of single testers, in my experience doing the scripted testing => exploratory testing migration requires that people understand what they’re doing, the limits of what they’re doing, what they could achieve if done otherwise.
We need to understand why they have been working in that way… maybe they were told to do so, or simply they aren’t aware that there are different and better ways.
Depending on the context, one thing we can try to do is to remove the inner details of scritped test cases… and start giving more and more freedom to the tester. At a certain point, people may reflect that they don’t need that guide anymore and that the guide/script actually limits themselves to discover valuable information.

To management, I think we have to show them the limits of scripted testing and traditional test cases. How they miss unknown risks.
Pairing with them could be something interesting to try out. In that case, I would not say “let’s write some test cases and then let’s try exploratory testing”. Instead, I would say “do you have some time? I woud love do to an exercise with you and use your expertise to provide feedback on something”… and then let that person explore somethiing and share thoughts with you. And then you could say “see, this is precisely the kind of value we can get with exploring… we learn, together… find problems, find ideas, gaps”.
It’s important to understand where that person stands (needs and concerns) and format the message accordingly. What I’ve seen is that people wants numbers and green/red flags because that gives them an easy way to deal with information.
Another way to look at it, is that with scripted tests, even if the system and risks were static (which they are not), we focus on finding bugs by looking at our expectactions.
With exploratory testing we have the ability to further improve the product… because we raise concerns, new ideas and we also find gaps. Checks won’t find gaps. It’s essential in a product to improve it by making sure we have there the things that matter and not just the things we’ve implemented.

1 Like

Every time is a good moment to explore :slight_smile: It’s not time that we waste, even if the product doesn’t change.
Remember that risks are dynamic (team is dynamic, context is dynamic, relevance of certain features is also dunamic, etc). Therefore exploring always provides value (unless you do exactly the same).
If you have a big feature, for sure you’ll need to explore it - and by explore I just don’t mean “explore after being built”; I mean it needs to be explored conceptually, discussed, explored whenever is being built and afterwards.
If you’re going to address a user story, then it’s good idea to explore it and at the same time implement some test automation for the happy paths, for example. Exploring complements automation (exploring also gives ideas for automation… and automation is also a way of exploring btw).
Whenever exploring we are walking in the unknow, and learning. If we postpone learning to somewhere is the future, the risk will still be there… the problem that it’s going to be harder to deal with it.

1 Like

Each person brings a perspective. As I mentioned, and quoting once again professor Cem Kaner,

Use “diverse half-measures”-- lots of different points of view, approaches, techniques, even if no one strategy is performed completely. – Cem Kaner, Black Box Software Testing

so each person has its own perspective, its own backpack of knowledge. Part of it may be overlapping, other not really.
Traditionaly, a developer has a greater backpack of knowledge in terms of the product, from a technical standpoint. Traditionaly, a tester has a greater backpack of knowledge in terms of testing.
Having developers and testers together, pairing, is for sure beneficial.
A conversation can start around the feature, from a funcional perspective, then go more to a technical side and back to the functional side.
In one of my teams, we have testers involved in mob-programming, being there to provide feedback and also do some guidance.

1 Like

Well, just to clarify: there are 2 apps :slight_smile:
What we call Xray (https://www.getxray.app/), that everyone knowns as such, is a test management solution that works inside Jira.
And now we have Xray Exploratory App (Xray - Exploratory Testing App) which is just a desktop app… that may connect with Jira+Xray, or you can simply use it without Jira+Xray :wink:

You can save your sessions locally, in a file. You can also export the session results to a .zip file, containining a PDF with all the notes and screenshots, plus the evidence as individual files.

If you do have Jira+Xray, then you can have some additional benefits such as uploading the results to Jira and also the ability to create defects right from the Xray Exploratory App.

1 Like

Yes, what you do whenever exploring is up to you. In that case you would probably record the whole screen, instead of picking up a specific application and/or browser.
What’s important, is that you have a charter that gives you the mission and some limited scope. Otherwise, you can be wandering around as James Whittaker mention.

Exploratory testing without good guidance is like wandering around a city looking for cool tourist attractions. It helps to have a guide and to understand something about your destination. - James Whittaker

It’s also better in terms of analyzing progress, because you can have a more limited and concrete scope that you can be able to provide a quality assessment.

1 Like

Timeboxing has some benefits

  • you have that allocated time for handling the charter
  • it can also somehow implicitly enforce you to look more at relevant things first
  • you may compare sessions afterwards
  • etc

I would say that using 30m or 45m is fine but this is something that needs to be tried in the team.
One of my teams is not really timeboxing though; that is working for us until now.

You mentioned about being fresh; I totally agree, we need to focus and defocus to foster different types of thinking. Making a pause, walk a bit outdoor, even washing dishes (which works for me), can provide you new ideas you can try when you get back to the keyboard :slight_smile:

1 Like