Ăine, big thanks to you and the Ministry of Testing for hosting, and we appreciate all the attendees dedicating their time.
To address the questions in more detail:
Q1. Tell us more about how u run ET. Is it done in pairs ? developers? testers? run in parallel? If so how do you plan the charters to be run in parallel? - Shalini
Regarding debriefing, from parallel running charters after 25 min, how much time do u allot for that in pomodoro? I understand the 25 min is for hands on testing. Then if one more session is there you need another planning as well. For me it is difficult to have 2 sessions and do all that in 1.5 to 2 hrs without tiring out as it is pretty intense. Any tips on share from your experience? - Shalini
A1. Working in pairs while performing session-based testing is found to be more effective, but the current practice is still often isolated, with async interactions (i.e. group planning, then individual exploration, then group review). Both the culture and the tool limitations in exploratory testing can be a factor.
All kinds of pairings among developers, testers, and business analysts can be valuable depending on the charter focus, the stage of the project, and, of course, teamsâ bandwidth (e.g. earlier on, you would definitely want to have the business side involved to maximize requirements improvement).
Running in parallel is more efficient, the challenge is ensuring both zero/minimal overlap and comprehensive project-level coverage. So, you need the team to agree on how the charter scope will be split by:
- an application area (e.g. registration, checkout, page X)
- a platform (e.g. desktop, mobile)
- a category (e.g. functional, usability/accessibility, performance)
Debriefing at the project level is, from my perspective, a bit outside of Pomodoro because itâs a group activity after you (and your teammates) complete the assigned portions of exploratory testing using the 25-minute blocks. The time allocated to that debriefing varies a lot depending on the team size, the number of charters being run in parallel, and the severity of findings. Two hours is not uncommon. âMini debriefingâ at the level of 1-2 charters (if they have related scope) could be around 30 minutes.
For cognitive load management, you donât have to do exploratory blocks back-to-back-to-back, as long as you have a way to keep track of your progress and evidence collected (like with Xray Exploratory App and Xray). Then, you can actually benefit from the context switch sometimes, where your work plan is something like: 25-min ET session â 5-min break â 25-min ET session â 10-min break â another task â 5-min break â 25-min ET session, etc.
Q2. Is this a free toolset? - GARETH DAVIES
A2.
- Jira â an issue and product tracking software from Atlassian with both free and paid versions.
- Xray - a test management plugin for Jira with a free trial then pricing based on user count.
- Xray Exploratory App - a free standalone desktop app.
Q3. In your blog you have said âThe fact that exploratory testing is mainly unstructuredâ but that sounds to me like there is a misunderstanding of what ET is - is this intentional to push people to use the tool instead of using charters etc? - Aj Wilson
A3. There is natural subjectivity to the exact definition of âunstructuredâ. I think there is a spectrum where, on the âcompletely unstructuredâ side, there is total chaos of âgo test thisâ (without any additional guidance), while on the âstructuredâ end there are scripted tests along with the culture that enables zero deviation from the steps.
To me, exploratory testing falls closer to the unstructured end of that spectrum because it doesnât have as much rigidity in the flow you should be following and as many formalized requirements before you start. But that doesnât mean the complete lack of planning with artifacts such as charters.
For the avoidance of doubt, Xray Exploratory App is not replacing the charter approach or session-based test management (nor is it providing a workflow without any structure). As you can see in the demo portion of the webinar, the starting point in Xray Exploratory App is a test or charter definition field. The tool will just help you collect evidence easier and transition it to a centralized repository faster, in a consistent manner.
Q4. How do you actually start and end ET? as it seems to be a never ending process? -
Rupal Baxi
A4. The session-based approach + Pomodoro determine the âdisciplinedâ 25-30 minute boundaries at the tactical level. At the project level, in many instances you will have external deadlines due to the release timing, so there will have to be difficult decisions about limiting the scopes of charters for a given iteration. Levels of risk/probability * severity/impact are typically a good guidance criterion.
Q5. Hi - I work in IT services company, having offshore & onsite delivery model, and multiple vendors are involved . How can we do exploratory testing in this context ? Also, enterprise applications are complex having internal & external integration with multiple systems - Balaji Thillai Kannappa Bharathi
A5. At the high level, the process wouldnât be much different from the âstandardâ session-based test management with charters, etc. Learning and knowledge sharing would be the key challenge, so there are two aspects to keep in mind:
-
Pair strategy - especially early on, you would want to pair onshore with offshore and, if possible, have cross-vendor collaboration. That ensures each time zone has the necessary foundation of info and helps get a very broad range of perspectives early on in the process.
-
âCharter libraryâ - you would need to track exploratory sessions consistently and store the charters, evidence, debrief reports in a centralized repository. That helps propagate all best practices and lessons learned among team members more easily.
With the number of systems and integrations involved, you may want to:
- leverage GenAI to help with test data for access
- split charter focus based on existing expertise (some testers/vendors may be already familiar with individual systems, others thrive more in exploring just the transitions from one to the other)
Q6. You had a showcase of Test Plans Metrics Report, from the documentation i see that this report is only available in Cloud version. Any plans to make this report available in Server+DC version? - Ivan Cizik
A6. This page reflects the latest status of feature parity between DC and Cloud.
Specifically, âTest Plans Reportâ in DC combines the aspects of Metrics and List reports from Cloud. By default, the âOtherâ category includes all non-Generic unstructured test types, including Exploratory.
Q7. Are there plans to integrate AI into the tool which could make context from similar issues available to get more test ideas on the fly while testing?? - Stefan Pryka
A7. We are definitely evaluating opportunities in the AI space, both for the use case in the question and others. As we progress in our assessment, we will keep everyone informed about any developments in this direction via official Xray channels.
Q8. Can you elaborate real world example of Exploratory testing? -
Muhammad Zafar
A8. The overall goal of exploratory testing is to complement scripted techniques to find the âmillion-dollarâ defects that are generally hidden behind the precisely defined scripted workflows, especially in complex processes that automation struggles with.
One of the common examples is exploratory testing for the cart checkout process, with the charter focus on âerraticâ user behavior (e.g. add an item, remove, add again, then edit cart quantity) as well as overall usability and accessibility.
Another example: for security testing, exploratory focus could be on the most recent threats that donât yet have thorough automated test coverage, and more specifically on uncommon combinations of those threats.
Q9. Do you have statistics on how much value ET derives and what kind of defects/bugs/ issues identified with the same? - Rupal Baxi
A9. The value of ET can be somewhat subjective and hard to define. If you are considering the ROI perspective, treating exploratory tests in a way similar to other types (e.g. issues in Jira with consistent metadata) would, in theory, allow you to apply the same ROI calculation based on time spent, quality achieved, etc. However, we would not recommend that approach as it may be too limited given numerous intangible benefits of exploratory testing.
When it comes to tracking defects/issues resulting from exploratory testing, with Xray and Xray Exploratory app integration, for instance, they will be linked to the test issue in Jira/Xray, which then will allow you to check them in the Traceability report. Once the tracking is established, you could try to approximately estimate the value of ET by putting a certain number on each âlesson learntâ depending on impact, priority, etc.