What is the consensus on AI being used to test a work item against it's acceptance criteria and verify it's implemented correctly?

Discussions
, , ,
1

Hi all,

I wanted to get some thoughts on using an AI agent to autonomously test work items (e.g. Feature, PBI, User Story, Defect etc.) directly.

This is not about using AI to create test scripts or automated tests. This is about testing a work item that has acceptance criteria defined, but we’re using an agent (by way of a skill and Playwright CLI) to verify that the work item meets its acceptance criteria instead of a human tester performing that assessment

My current company is going down this route - they also use an exploratory testing skill to explore a website, although it can be thought of more as a bug hunter as opposed to any sense of it verifying functionality works as expected.

I’m slightly hesitant on this approach - it does seem interesting and promising, and not something I’m dismissing out-of-hand, but the way it’s been implemented is that you’ll tend to get an incredibly lengthy output commentary rather than anything “tangible”. The commentary can be 1000s of words, and I’m skeptical other QAs are actually reading it before signing-off (which the agent essentially does anyway by moving the work item to done).

The agent will often suggest tests to add - it prioritises unit tests over other layers, and will need to justify any tests added at higher layers. I understand this approach in theory, but I’ve seen instances where I don’t believe the test gap can be fully or even partly addressed by unit tests alone, and an integration or e2e test would also be valuable, and in some cases required in order to catch that particular type of bug in future.

Full disclosure - I am a heavy AI user (Copliot and Claude) and I’m very pro-AI, but still tend to adopt a fairly cautious approach in that I need human judgement involved and don’t just want to press a button and have AI do everything while I go and grab a coffee.

Would be interested to hear other’s thoughts on this.

2

Well letting AI explore burns a lot more tokens than expected. Also, there’s always a product decision to do something in a particular way which it may not understand.

I did give that a try but I still feel a human’s exploration is better. The AI can build my notes, clarify any complexities for me (like monitoring the behaviour in comparison to network calls) and write automated tests.

I’m not sure if an acceptance criteria can be 100% well written such that the AI can fully understand what is it doing, if so, the developer may as well just have done a job so perfect, you don’t even need any tester to look at it.

3

A couple of things that I would be concerned about using this approach:

Who defined the work item? Where did the acceptance criteria come from? If the acceptance criteria is the last time in the process that a human is involved (I assume your company is using AI for dev too), they must be perfect. Any issues with the acceptance criteria are less likely to be challenged by AI.

Who are the target users of the website? Are you prompting the testing skill with personas? I don’t know how good AI models are at simulating doing something stupid or unexpected. Will they get better or worse at this as they become more powerful.

I do think there’s a place for it, but as a stage in the process. I imagine it would work well for reopening an issue that clearly did not meet the acceptance criteria, but I would be extremely reluctant to give the agent the power to mark the work item as done.

As long as we’re building software for humans, I think a human-in-the-loop approach is going to result in a better end product.

4

i’d separate agent execution from release authority. let it test each acceptance criterion, but make the output compact: pass, fail, or uncertain; the exact browser path; the observed final state; and a trace or recording someone can inspect. thousands of words is a failed interface.

before an item moves to done, a human should review the uncertain cases and risks the acceptance criteria may not express: permissions, state carried from earlier actions, integrations, accessibility, and destructive side effects. until you know the agent’s false-pass rate, “ready for review” is safer than “done.”

for regression coverage, choose the lowest layer that preserves the bug mechanism. if the failure depends on wiring or browser state, unit tests alone are the wrong evidence.

5

I’ve seen a lot of automated workflows, from intake to exploratory testing, across 100+ engineering teams. And I have to admit, human behavior there is usually no different from what you describe.

There’s no direct question here, you want opinions and stories. OK, here we go.

A topic this broad probably deserves an opus, but I hate writing long texts and prefer max information density. Sorry folks for the style below:

- You say the idea is promising, but the problem is “the way it’s been implemented”. I disagree. The way is not what will define the success or failure of this initiative. I could implement what you described in 5-7 different ways, all of them would work as you describe, and none would solve the pain: the quality of your product won’t increase.

- Coding is a deterministic task, QA in my opinion is not. For code we have tons of approaches, tools, validators, methodologies, you name it, to ground the LLM and get verifiable results, and even there we can’t fully rely on AI. So no, I don’t agree with Boris Cherny that coding is “practically solved”. In my experience only simple tasks can go through “dark factories” today without human intervention. I haven’t seen a successful implementation of QA agents that really verifies quality

- I’m not saying it’s impossible in the future. But today it looks like quality vs money to me: if you want quality, the cost goes not to the moon, but probably to Saturn :slight_smile:

- A strong QA, armed with methodology, experience, tools, previous results, gut feeling and a sense of psychology, can’t be reproduced by a bunch of agents, workflows, routines or graphs. I’ve built quite a few “job replication” agents in different areas, and QA is the one job I’d politely pass on :slight_smile:

- Even with 10k+ passing e2e tests (or 100k+, pick any number) you can’t be sure a release is safe to deploy to prod. In most cases you don’t know what code coverage those tests actually give you, or whether they mitigate all the risks of a new feature. And every new feature needs new scenarios, tests and especially data, which is so time-consuming, even with AI, that I’ve rarely seen SDETs fast enough to cover everything and use e2e on the QA env to sign-off release

- You mentioned agentic testing against acceptance criteria. Interesting: in how many cases does your env actually have the data for such testing? I’m not talking about the ticket where a button should be moved and recolored :slight_smile: Curious, because I deal with real enterprise apps (100/10k/100k+) users, tons of dependencies and complex data flows, mostly transactional or document-oriented systems. In my situation AI-driven acceptance/exploratory testing is a waste of time and burned money

- From what I see around, quality work is the last thing people try to fully automate with AI. A non-deterministic instrument for a non-deterministic task sounds like magic to me :slight_smile: If you already have measured results showing it’s possible for reasonable money, please share, it would be really interesting!!!

- Unit, integration, API, e2e, manual: rebalancing the pyramid is kind of art, at least it is how I see it. But with AI this deep in the SDLC, I think someone will come up with a good methodology backed by math

- Even when acceptance criteria are written by an experienced BA, from what I’ve seen, all of them assume the “receiver” already has the context in which those criteria make sense. Is that the case for you?

I hope people with hands-on experience can answer some of the points above
Disclaimer: this is my humble opinion, based on my experience, my education, my circle of collaboration, the universe I can reach
P.S. Looks like this post is longer than KB page I recently wrote :slight_smile:

6

If acceptance criteria is known very well and the AI has very good oracles in relation to risks to that acceptance criteria it will likely be able to do this to a fair extent, in some cases better than a tester not fully up to speed on that specific risk. Accessibility risk for example, good oracles are in play and some testers have not studied this deeply yet so an agent may do better.

Exploratory testing is very different though, it is used specifically when unknowns are in play and to be done well it requires that primary skill of active real time critical thinking and analysis. That is the key question to consider, the value of the real time critical thinking and can an agent get close to that.

Before agents this was more straight forward. Automation - machine strength stuff on known risks. Testers - active critical thinking and analysis during testing when unknowns are in play. Do agents change that equation?

Agents can though crawl your app real time and likely do a level of real time analysis, good oracles it will pick up issues, missing oracles, ambiguity, nuances, unknowns not trained in then its attempts to simulate critical thinking could likely fall short. As others suggest this involves a looped investigation so could also get costly if you put this into CI for example. BearQ for example seems to suggest they are close on this one but even advocates of this tool will not suggest it as a substitute for active critical thinking.

I reached a compromise I am happy with, machines for the known very well stuff, some agentic crawling as a first round scan to pick up on some basics to investigate further and then a paired approach with goose and playwright-cli with rapid experiments and back and forth interactions so keeping my critical thinking strength in the thick of the action but letting the tools give me more visibility and sometimes run machine strength experiments. This approach also keeps token use lower, team knowledge levels high and I having zero qualms on taking accountability of what the machines do under my watch.

I like this compromise, exploration is my primary fun factor and motivation in testing so I have a bias not let machines have all the fun but it also works well, my testing has become more efficient, I can do deeper testing and it also has expanded my remit with much easier root cause analysis, solution recommendations and smallish safe fixes.

7

One thing we’ve found interesting is that the hard part isn’t necessarily getting an AI agent to execute the acceptance criteria. It’s preserving the relationship between the requirement, the tests derived from it, the evidence from execution, and the defects discovered along the way.

If the agent finds something that doesn’t meet the acceptance criteria, that result is much more useful when it becomes part of the same traceable testing context rather than just a long AI report.

I also agree with the distinction between “ready for review” and “done”. Autonomous execution and release authority don’t necessarily have to be the same thing.

8

Coming at this from the developer side, one rule has worked well for me: the agent can say “this should now meet the criterion”, but it can’t be the one that records the pass. Its result counts as “ready for retest”, and a person confirms it before the item is done.

Two reasons. First, an agent checks the criterion as written, and most of the issues I’ve seen slip through were in what the criterion didn’t say: state left over from an earlier step, a double tap, a slow or dropped connection. Second, when AI wrote the code and AI checks it, their blind spots tend to line up.

+1 to @dwaynesamuels on keeping the output compact. I’d add one thing: record it per platform. An agent passing a criterion in a desktop browser tells you very little about the Android or iOS build.

9

agreed. i’d treat platform as part of the test result, not metadata. the same criterion can pass on desktop and fail on mobile because the path, permissions, storage, network behavior, or native handoff changed.

have the agent return one small row per platform: build, starting state, exact path, pass, fail, or uncertain, final-state readback, and the first mismatch. then the human retest can compare outcomes instead of treating one desktop pass as universal.

if AI wrote the change and the check, add at least one independently defined negative case before marking it done. that gives the reviewer a chance to catch shared blind spots.