AI Is Boosting My Exploratory Testing, But How Do I Justify The Cost

Discussions
, ,
1

My company has been very supportive of us experimenting with AI and bringing it into our day‑to‑day testing workflow. But as AI usage (and pricing) keeps rising, we’re now being asked to justify our token consumption, especially when the numbers start looking large. For automated test generation, that’s easy: you can point to tangible outputs.

But for non‑automation work, it’s harder. I’m using Claude more and more as a testing assistant, particularly during exploratory testing. I bounce ideas off it, ask it to calculate sensible coverage permutations, generate test ideas, help me reason about risks, or summarise what I’ve explored so far. These interactions genuinely improve my thinking and speed, but they don’t produce a neat “deliverable” you can attach to a ticket. They’re more like having a second tester in the room. I imagine others are in the same situation.

Is anyone else using AI this way, and how are you justifying the token usage to your organisation?

2

i’d justify the spend by sampling decisions it changed, not counting prompts or generated text. for the next 10 exploratory sessions, log the risk you were testing, where Claude changed your coverage or next move, time spent, and whether it exposed a bug, ruled out a risk, or prevented duplicate exploration.

compare that with 10 similar sessions without it. if the assistant produces faster useful decisions or broader risk coverage after review, you have an economic case. if all it produces is nicer notes, you probably do not.

one guardrail: keep the tester responsible for the oracle. Claude can widen the search, but the browser behavior and resulting decision still need human verification.

3

Thanks, that seems like a really sensible approach that gets to the heart of what I’m trying to measure. I’ll give this a try, it will be an interesting experiment

4

Do you know what your token usage is? If not first try and establish what your token usage is, I’m no expert but those things do not sound like power using token activities.

If for example you were running it as agentic crawlers like a user running your app picking up common bugs on every CI run that would be a very different thing as it likely goes into process loops in many cases but you seem to be using it hands on, rapid back and forth loop in an interactive way which I doubt would be heavy usage.

Check if you have been flagged as a heavy user first of all, if so try and establish why, also compare to other users in the company just as an information only reference point. If you are part of a company group usage, a fixed monthly fee or if they you individual purchases or paying by usage that could make a difference.

I use it in a similar way though I also use playwright_cli for exploratory experiments, rapid short feedback loops and that back and forth interactions, I also find this model as a booster and much quicker at root causing at code level when I find something alongside recommended fixes.

Justifying a monthly capped licence may be easier than justifying token usage.

5

What metrics do the company care about? How do they judge the value of exploratory testing? If they have something, I’d aim to link the AI usage cost to an improvement in those metrics. If they don’t I think you’ll just have to argue your case - if you’re increasing your speed or coverage by x%, then anything less than x% of your salary seems like a reasonable cost.

Another thought - If your devs have used AI to increase their output if follows that QA will need to increase their output to match - that may be enough justification on it’s own, and you can just explain how AI makes you more efficient.

6

You make a good point about understanding whether I’m actually a heavy user before worrying too much about justification. To be honest, I’m a bit in the dark about how other people are using Claude and what a “normal” level of usage looks like, so it would be useful to find that out.

I think I’ve just hit a standard cap where some justification is needed before they’ll raise the limit, which is fair enough.

Part of the problem is that, when I was working as a solo tester, I was getting Claude to create the test plan and write tests within a single chat using Opus. It worked well, but it was quite token intensive.

I guess in a larger organisation I’d be linking things to test management systems, and the work would naturally be broken down into smaller chunks. I’m now looking at ways to use tokens more efficiently, for example using Opus for the planning work and then Sonnet to help generate the actual tests.

I’m interested in how people justify the token usage and demonstrate the value AI is providing. It’s similar to questions I’ve asked here before: without a baseline, and without any metrics showing how long testing took before AI was introduced, it’s difficult to compare.

I suppose I need to pull together some concrete examples where AI has genuinely sped up the process or improved the quality of my work.

Thanks for the tip about using Playwright CLI for exploratory work. I hadn’t thought about that approach. The majority of my work is desktop testing on a WPF application, so it’s not as straightforward to use desktop automation frameworks in the same way. It could be useful for my web application testing though.

7

Claude Opus (other vendors do exist, but are subjectively rubbish) is not that expensive (if you are in the west). Engineers far too often get asked to justify an expense, because manager are just not doing their job well enough to detect the productivity boost. So if we lie to them, it’s…

(Disclaimer, I’m loving it, recently got it too) Why? Because although I’m not using it to write tests, it’s still so useful to have a partner at the company who costs about 1% of my salary, and is good at continuously reminding me what I was doing what I’m doing next, and generally being far more helpful than the coffee machine.

8

You are getting it to “generate”, you need to get it to “do” the work as well.

Part of your costing problem is that management now believe that using an AI tool can make up for them not spending time building testability into the product and for time spent finding and creating test frameworks. I know that sounds harsh, and will not go down with managers well, but I suspect you can find a more polite way to word it in your specific context.

Basically you seem to already be on top of things, you are choosing when and how to use the tool, and yes you will need to be using Opus a lot to set up the frameworking, which is why it is an expense the business delayed in paying until AI came along. The AI companies are counting on customers like this who have time-to-market opportunity cost profiles like their own.

Or, relate this story:

"Finance rejected the $18 renewal for our company domain.

Reason:

“No documented ROI.”

This is the domain used by our website, email, customer portal, VPN, SSO, invoices, and every employee address.

They asked IT to provide a cheaper alternative.

OPTION B:

Cancel the domain.

Issue 900 new email addresses.

Update 84 SaaS integrations.

Reset 11,000 customer logins.

Replace every email address on every contract, invoice, business card, and vendor account.

Notify our bank.

Reconfigure SSO.

Buy a different domain.

Estimated migration cost: $640,000.

Annual savings: $18."

The bonus to this story is that nobody had to learn about DNS, and it’s true for using any tool that helps you work better, whether your boss notices it, or not.

9

I’m out of work right now, so this one lands a bit differently for me. I still develop and write automation day to day, and I’ve paid for Claude out of my own pocket, absolutely worth it, plus Copilot too. Both self-funded, nobody expensing it for me.

I think it’s worth every penny. I hear from a colleague who’s a dev that he’s using Claude daily through his company with barely any cost pressure at all, there was definitely a stretch where everyone seemed to be racing to hit the token limits rather than watching the bill.

What worries me is where this goes next. As AI companies go under (some will eventually I suspect), there’ll be fewer of them left, and the ones that remain will keep raising prices. I think eventually it tips toward only companies being able to afford the newest models, and individuals, especially anyone out of work and paying for it themselves right now, end up priced out or stuck a generation behind.

10

@checkout_champion - Without divulging too many company details, can you give us 1 or 2 examples of how AI helped you in exploratory testing? Perhaps this could be a separate post if you wish to talk about it. I am curious to know how much of the AI answers are new and unique ideas VS things the tester did not know about testing, but should probably know.

11

Me: “Hey Claude write me a script to time the first and subsequent runs of this application c:\pathtoexe”

Claude writes a 60 line script, that not only makes good guesses as to how many times to run the app, but also prints out a mean min and average , print the 1st time cold-start value, and makes it parameterizable so I can change anything without touching code, to let me run any other apps I might want to trial as well, nothing was hard-coded, all with defaults so no need to think Literally an exploratory test in 2 minutes that would have taken me closer to an hour of pontificating over.

12

I, as a QA Engineer, spend about $1K monthly (usually a bit more). I use AI for everything - literally all activities but my setup/agent is more or less optimized in terms of token usage, and I justify nothing because I can work (and I worked) without AI, so if corpo managers decide to cut the budget and give me $500, I’ll spend it. Honestly, this gives me nothing, but the corpo benefits from me using AI. If the corpo doesn’t want me to use AI and speed things up, so ok, I’m fine with that; I would say working without AI was mentally better

13

@conrad.braam Yes, I have seen its useful for generating scripts and utilities to speed up both exploratory or non-exploratory testing. Do you have any examples in which it provided new or interesting testing ideas or risks that you did not think of?

14

Only anecdotally, it gets things wrong often enough that it shows me possible negative test cases I would never have thought of. the rest of the time I am trying to get it to speed up writing little test tools, like utilities that automate anything I want really.