Has Anyone Replaced Manual Testing with Agentic AI for New Feature Testing?

Discussions
,
1

Hi everyone,

I’ve been exploring agentic AI solutions in QA and I’m curious about real-world experiences.

Has anyone here significantly replaced manual testing (especially for new feature testing, not regression) with an agentic solution?

I’d love to hear:

  • How much manual testing are you still doing today?

  • To what extent do you trust the agent to generate test cases and accurately determine pass/fail results?

  • What worked well, and what didn’t?

Trying to understand where these tools truly add value versus where human testing is still essential.

Thanks in advance for sharing your insights!

2

As a first step its worth creating some sort of matrix that lists the testing activities and values and then trying to classify whether they lean towards mechanical or human strengths.

A lot of teams did this previously but when you look at the market a lot of other teams were sort of having their humans doing quite a few activities that leaned towards mech strengths.

What does your manual testing look like?

If it leans towards test cases, scripted testing and testing to verify its likely AI can both replace and accelerate a lot of this.

This is not a model I follow which means my bias will often place things like test cases of lower value so AI generating this its often yep good enough. Similarly if I use something like playwright agents to automate this it fits into a bit extra coverage that for me is good enough. Hard core test case writers and deep automaters though will challenge AI as they are often doing very domain specific coverage and complexity increases risk for AI.

If it leans more towards a learning of the product, discovery, investigation and exploration of risk for now this still very much fits in human hands but with AI empowering potentially deeper and broader risk investigation.

The latter model have a look here How are you actually using AI in testing right now? - #2 by andrewkelly2555

Now there are tools offering exploratory testing with multiple agent running. My take useful for large teams on enterprise level products but capability needs to be clearer, they remain in my view closer to crawlers and an expansion of known risk coverage for now and not the human strength stuff testers do.

There are a few interesting areas. Lets say two areas which say your testers have limited experience in, security and accessiblity for example. These AI tools will often outperform an average tester on their own but again become powerful tools in the hands of someone advanced in those risks.

If human testing drops on human centric products its high risk this is acceptance of lower quality and less innovative products and that lower bar is accepted in exchange for faster more autonomous coverage.

3

I really appreciate this detailed explanation. Most of our work today is exploratory testing, where we focus on understanding product behavior, identifying risks, and validating user flows rather than relying on predefined scripts.

4

Interesting discussion. I want to share a different angle we’ve validated in practice.

There’s an approach that sits between fully manual and fully agentic: AI pre-annotates UI elements, humans review and correct the annotations, then deterministic test scripts are generated from that. Once generated, execution costs zero AI tokens — it’s just standard Python hitting precise coordinates. No LLM in the loop at runtime, no flakiness from model variance.

This works. We’ve proven it on real applications.

But honestly, it’s not something every QA team can just pick up and do themselves.
The annotation pipeline needs specific tooling. The visual recognition models need
training data. Getting the precision right so tests survive minor UI shifts takes
real engineering effort. For most teams, the ROI only works if someone else has
already built and maintained that infrastructure.

Which brings me to what I’m genuinely curious about from this community:

What does your team actually want — better tools to write tests, or reliable test
code that just works? If someone handed you deterministic, maintainable scenario
scripts covering your critical flows, would that solve the problem? Or is the process
of writing tests itself where the value lies?

And a practical question: once you’ve achieved coverage for your core user journeys,
how much churn do you typically see? Is maintenance a constant battle, or more like
occasional updates after major releases?

Would love to hear how others think about this tradeoff.

5

I think you highlight some of the core challenges as a whole, what problems are you trying to solve with AI.

“What does your team actually want — better tools to write tests, or reliable test
code that just works? If someone handed you deterministic, maintainable scenario
scripts covering your critical flows, would that solve the problem? Or is the process
of writing tests itself where the value lies?”

For me none of that would solve my testing challenges, useful yes but its focused on known risks so say 10% of what a hands on tester focuses on, their strengths lie in the unknowns. Most apps are still for this point at human centric and as such have never really been deterministic even when guardrails are in place.

I added this description of testing to another thread recently which may help explain why my goals could be very different.

“Testing that highly technical tool loving activity that emphasises learning, discovery, investigation and experimentation into product risk. The one that embraces the currently unknowns, finds comfort in ambiguity, nuances, empathy and real world context. That takes a holistic whole lifecycle view of testing and applies it from day one.”

@probe_runner What you may want from AI is likely very different from I, the tools I am looking for are of data and information based, give me more visibility on what’s happening as I test, maybe I miss some odd api responses as I test, is there something in logs that give me more insight, what experiments can I do next.

An automation engineer for example may have goals much closer to the things you are suggesting but potentially a lot less so for a hands on tester. Its really important and of high value you raise this and its a great question, “what does your team actually want from AI?”. Its not going to be the same for all testers and your points are very valid, in particular token use in agentic will be interesting.

6

Good point — and actually what I described only tells half the story. The UI interaction is just the trigger. The real gate is underneath: every API call during a scenario gets recorded, and on subsequent runs the responses are compared automatically. So it’s less about “did the button appear” and more about “did the backend behave the same way.”

That might actually overlap with what you’re looking for — visibility into what’s happening under the hood as tests run. The difference is it happens passively: you run the scenario once, and from that point on any API drift gets flagged without extra effort.

Beyond exact response matching, things like 5xx spikes, latency drift, or unexpected status code changes across runs get surfaced automatically — the kind of odd API behavior that’s easy to miss during manual exploration.

Your point about different testers wanting different things from AI is spot on. For hands-on testers, this kind of passive regression net might be more useful than generated test scripts.

7

This is sort of the thing I want an agent to pick up on as I explore and run my own experiments, as you suggest it can be missed but an agent should be able to pick this up and highlight it. It’s on my list to look at say playwright cli to see if I can get it to do this in the background as I test.

8

I use ito.ai for QA as it actually runs the code (unfortunately takes a lot more time than regular ai code review tools). It’s a very human like tool and it even provides screenshots for every report

9

Thanks for sharing this interesting insight.
That may also come from a slightly different angle in my case.

When I posted this, I was actively evaluating tools and running experiments to see how far agentic QA approaches can realistically reduce manual effort in practice. I also have some real concerns about the idea of fully replacing manual testing with agentic solutions, so I wanted to measure that gap rather than assume it.

That’s also why I was curious about how other teams are actually working today — not just the direction they’re aiming for.

I agree with your point that “what does your team actually want from AI?” is a very important starting question. But I think it naturally leads into more practical discussions like cost, scale, and especially token usage in agentic workflows — because that’s where the trade-offs start to show up in real systems.

So for me, the interesting part is less the framing and more how this translates in reality:
how teams define value, what they’re willing to delegate, and where they still insist on keeping strong human involvement in the loop.

10

Yes, cost/scale is where the trade-offs become unavoidable in real systems — agree.

The split I keep coming back to: AI for generation (writing tests, turning a spec into scaffolding, suggesting edge cases) is genuinely useful — I treat the output like a junior engineer’s first draft, always reviewed. AI at execution — an agent deciding what to click, what to assert, what to retry on every run — is where the token cost compounds and determinism collapses. A test that runs deterministically once it exists is something you can put in CI and trust. A test that asks an LLM “what should I do next?” on every run burns tokens every time and can never quite be reproduced.

So my honest answer to “where do you keep humans in the loop”: humans review the generated code once, then the test runs without AI at all.

The agent does the hard up-front work; the runtime stays cheap and deterministic.

What’s the breaking point you’ve usually seen on the execution side when teams keep an agent in the loop at runtime — cost, flakiness, or trust? Curious which one tends to surface first.

11

This still looks like agentic AI for automation and less so for replacing the human strength activities that focus more on testing to learn and experimenting with risk hypothesis where variation every time is high value rather than the deterministic stuff that test cases and automation has previously covered where repeatability is highly valued. This is valuable too but I thought the topic was more focused on the replacing the human strength activities rather than the things we already deem as mech strength activities?

12

I agree with this distinction.

I don’t think “replacing manual testing” should be the goal by itself. The goal is faster feedback, better risk coverage, and lower cost.

Agentic AI will probably start with the mechanical/repeatable parts because that is where it is easiest to measure value. Then, as the tools get better at observing behavior and generating useful variation, they may naturally move further into exploratory testing.

But replacing human judgment and learning is a much higher bar.

13

I agree, replacing human value and strengths should not be the goal.

I’m sometimes reminded of the meme, “AI can spend time with your kids so you are freed up to spend more time on cleaning the house”, wrong in so many ways.

I love discovery based testing and at some point we may well ask why should robots get all the fun.

On a side note, the lower cost will be an interesting one to keep an eye on but I’ve always found lower cost goals in testing to be a distraction and often a fallacy.

14

Andrew, the “AI handles the fun stuff so you can clean the house” framing lands hard — sharper than any whitepaper I’ve read on misaligned automation.

I should be upfront on where I’m coming from: developer by background, and my scope is narrow — UI-driven API regression. Concretely: a person (or recorded flow) drives the UI, every API request and response gets captured, and subsequent runs diff against that baseline. It came out of necessity —
deterministic UI automation was a persistent pain point for my QA team, and what worked for me was sidestepping selector fragility with visual coordinates: positions captured once at annotation time, replayed positionally rather than re-querying the DOM each run. That alone is what made UI playback deterministic for me.

The AI piece is purely scenario discovery: I don’t have the bandwidth to hand-author the breadth of flows I’d want covered, so I need something to surface candidates I can then curate. For me, that whole stack is the gate that makes refactoring safe — and the bar’s gotten higher
fast in an AI-coding world where the diff didn’t necessarily come from someone you can ask what they meant.

You’re right to push on the cost framing. I flattened three things into one sentence. Cost as a goal runs into the same Goodhart problem most quality metrics do — chase the number, lose the thing the number was supposed to track. The piece I should have separated out: when an agent runs at execution time, every CI run burns tokens, and uncapped token cost is what keeps you from running tests often enough to get the faster feedback part. So cost-as-constraint, not cost-as-goal — the determinism argument is really about predictable cost enabling frequency, not about cheaper testing being better testing.

On the fun question — I’d frame the division as: robots get the tedium, humans get the judgment. The “fun” in exploration is the cognitive surprise itself — the “huh, that’s weird, why does it do that” moment. An agent flagging an unexpected 5xx during your session can hand you the surprise faster,
but it can’t have the surprise. That part doesn’t transfer.