Has anyone used an AI agent as a tool to help guide them to areas where they may want to do deeper exploratory testing?
2 Likes
Weâve actually been exploring this idea in our tool â using AI not to replace exploratory testing, but to supercharge it, especially during things like bug bashes.
The way we see it, AI can throw out a bunch of interesting scenarios, âwhat if?â questions, or even oddball challenges that testers might not think of right away. Itâs not about the AI doing the exploration for you â itâs about sparking your thinking, breaking you out of obvious paths, and giving you new angles to investigate.
Kind of like having a really curious teammate whoâs constantly asking, âBut what if the user does this weird thing?â or âWhat happens if the server hiccups here?âemphasised text
From one of our protoypes:
3 Likes
I struggle to see how this would be useful. In 30 years of exploratory testing, I have literally never run out of ideas for tests on any of the hundreds of projects I have worked on. I always run out of time first.
If testers run out of ideas, itâs because of the almost total absence of good exploratory testing training. Apart from the AST, BBST and RST courses, thereâs pretty much nothing. I developed a pretty good 4-day course but I donât deliver it anymore.
The result is that the quality of exploratory testing is pretty dismal. We trained a handful of seriously good testers, but I was always astoundingly unimpressed with the quality of contractors and interview candidates.
But the solution isnât to use AI as a crutch because it can only provide some ideas, most of which wonât be useful. The tester still needs to do the iterative cycle of test design, test execution and learning, and AI wonât help them get better at that. The solution is for testers to take their personal development more seriously. Go on the courses, read the books and learn to read code.
5 Likes
I think that I agree. I think a responsible tester can use those tools, ethical concerns about data sourcing and energy usage aside, if they are indeed informed and responsible. Your point about the quality of testing being dismal.. well, yes, and that I suppose directs me to a similar point.
My worry about any tool is that it serves as a replacement for capability and thought. Testing software is centred around human beings because software and users and the business exist in a series of complex and complicated social systems. We evaluate based on information specific to a company, user base, tech stack, team, global economic situation, competitor behaviour, whatever we care to know about. I can ask AI for ideas on how to test something, but it would be devoid of the context of my software and users, and I can only spend so long explaining what these things are and why they might matter to a tool. That accumulated knowledge, tacit or otherwise, is what makes a tester so powerful. Combined with practiced skill this is what makes humans such powerful explorers, and the use of tools is guided by that same knowledge and skill.
Itâs quite insidious, because AI can generate ideas. And testers can generate ideas. So they look the same. But testing happens within testers and their internal structures and models, and should leverage a lot of understanding about the needs of test clients and a myriad of other contextual factors that an AI simply does not have access to. The goal would be to find factors it has access to that we do not, and use it that way. Perhaps simply as a safety net, in the same way a checklist might be used.
Testing is a social science but a lot of the material written about testing suggests that itâs mechanical, procedural and formalised. That means that AI systems trained on the vast fields of nonsense written about what testing is, can do, and is, all feel likely to repeat that nonsense. If an AI suggests that I use faulty metrics (and thereâs a lot of things written about how to employ metrics that donât work) then itâs giving me terrible advice. So it requires a certain amount of understanding to be able to reject the tool when it gives us faulty or damaging ideas - making it more of a tool for experts, perhaps.
Iâd look to helpful lists if I wanted to generate ideas. The HTSM is a good way to add to ideas that the software itself brings up as we explore it, for one example. Iâm not sure how AI can fit in here - without knowing the software it canât define ideas by product area, so theyâd have to be more generic. It could help to create connections between facts about the software and testing concepts - so if I told it that I was creating something for a specific domain it could give me facts about that domain related to testing it. As we adjust the testability of the software by accumulating knowledge, an AI could be part of finding and collating those facts - although theyâd have to be checked if theyâre specific and important facts.
I think AI is likely best used as a slightly uninformed brainstorming partner for specific cases. To be able to feed in context about who is testing, what is being tested, what risks youâre testing for, how you want to go about it, and/or the oracles against which you are evaluating you may be able to find connections to other concerns, or scenarios that make sense. To flesh out techniques, help find holes in coverage, and so on. When we have those then a tester can decide how to apply resources and if theyâre worthy of concern. Whether or not thatâs worth the cost is another concern.
3 Likes
Thatâs an astoundingly good analysis.
3 Likes
Iâm trying really hard not to be a hater here, but why would someone trust that your tool is worthwhile when you couldnât even be bothered to write your comment yourself? It is painfully obvious you used an LLM for your entire commentâŚ
AI can be a practical tool, but it becomes extremely worrying when people outsource even the most basic writing tasks to it.
I just checked the comment on GPTZero and itâs come back as human. It felt human to me too, I wonder what Iâm missing.
What parts make it feel LLM created?
Two markers stood out to me - first, the use of Em-dashes. Yes, people do use them in their writing, but most people donât know how to do them from the keyboard. LLMâs however use them very liberally. This is a known marker of LLM output at the moment, though I can see the various providers putting in patches to try and remove this âtellâ that content is from AI, so it may disappear soon enough.
Secondly was in a later paragraph where it fell into a very common LLM pattern, âItâs not about X, itâs about Yâ. Again, of course this is something a human might write, but the combination of the two flags stood out to me.
Oh, and thereâs the âemphasised textâ left at the end of the final paragraph, not sure if that was an artifact from MOT formatting options in the posting field getting broken by accident, or if itâs a leftover instruction that wasnât processed properly?
Other posts from the user also follow the current LLM flavour - lots of lists with emojis for dot points. Once youâve seen it a couple of times, it stands out like a beacon when you next see it.
Look, I could be completely wrong, but I donât think I am. And I think the ongoing outsourcing of our creativity is worth calling out from a quality perspective - as a potential customer, I am instantly turned away when a product looks like itâs being presented (and thus, potentially put partially together) via AI. It doesnât matter how good the end product is if the marketing steers people away from even trying it in the first place.
Anywho, just my 2c⌠