Is agentic AI testing real?

Discussions
, ,
1

Hi,

The company is pushing quite strongly for a major shift from manual testing toward semi-automated testing. The main process they want to implement is the following:

A Manual QA creates TMS test cases in a format specifically prepared for automation. The test is then sent to Claude, where a dedicated skill is used to automate it. The resulting automation is added to an MR and reviewed by AQA.

However, I have some concerns about whether this approach is actually suitable for our project.

The domain is quite complex, with constant or partial interaction with the canvas, and the project itself is highly complex due to its iGaming nature. Each slot/game is essentially a separate project within Playwright, and the games can differ significantly from one another - from relatively minor differences to completely unique features and mechanics. In some cases, a particular game may also use its own unique WebSocket protocol or message structure, meaning that even the communication layer can be specific to that game and cannot necessarily be reused across other slots.

Given this, I’m questioning whether automating tests in this way is actually worthwhile. I can see the value in this approach for specific, well-defined areas, such as API/WebSocket testing or very basic UI scenarios that are common across multiple slots, since their structures and templates are relatively similar.

However, trying to apply the same approach to more complex and game-specific scenarios seems much more challenging. Important factor is that none of the Manual QA team members currently have programming or AI-related knowledge, which makes it difficult for them to properly understand, validate, and maintain the generated automation.

Overall, I’m concerned that we are trying to apply an AI-driven automation process to a project where the tests themselves are often highly game-specific and technically complex. In this environment, generating an automated test may be relatively easy, but ensuring that it is correct, reliable, maintainable, and actually provides value could be significantly more difficult. This could potentially result in a large number of automated tests that require substantial effort from AQA to review and maintain, while providing relatively little benefit compared to the effort invested.

Have you ever faced a similar situation?

  1. Can someone realistically validate and maintain AI-generated Playwright tests without programming knowledge?
  2. How would you convince a Lead or Manager to reconsider an automation strategy that you believe is fundamentally flawed?
  3. If you were in my position, would you try to prove that the approach works first, or prove that it doesn’t? How would you design the experiment?
2

Yes, something similar is happening all around me.

My question is: What is the real benefit ofvthis approach?

I am not seeing any progress with this kinds of approaches, only pain and seffering for the services owners.

My two cents, keep using AI as a tool, helper, to speed up some tasks. Write YOUR code as your are “signing” it, and sleep with confidence :grin:

3

I think agentic testing is real, but we should be precise about what has changed.

The cost of producing test code has decreased substantially. That is not the same as reducing the cost of testing, or even the cost of maintaining an automated suite. An agent can generate a lots of Playwright code quickly; it can also generate duplicated abstractions, brittle selectors, unclear intent, and a larger suite than anyone can sensibly own. (I don’t trust self healing locators just yet)

Whether we are vibe coding an app or building automated tests, the engineering concerns remain: modularity, clear boundaries, maintainable abstractions, meaningful reporting, controlled test data, and a design that makes failures diagnosable. Those are architectural decisions, not code generation tasks.

I would add that the cost of test design has not disappeared either. We still need to decide which risks matter, what evidence would change our confidence, and which checks belong at which level. An agent can help explore, implement, execute, and summarize but “the tests passed” is only evidence about the scope and quality of the tests we chose. You also need more evidence when you let agents run tests.

So yes: agentic AI can make testing more capable and faster. But without good testability in the product and good engineering in the test suite, it may simply let us create automation debt at a much higher speed.

I’d add onto this that at work I’m moving us away from the assumption that a UI scenario test case written by a tester is what should be automated. We want automated tests to find failures at the lowest useful level of the stack where they are faster, more reliable, and easier to diagnose.

4

Yeah, that sounds logical. But the main scope of my work on the team is testing the integration of features into the pre-production environment, so I’m testing on a live environment without any mock data.

5

In my previous company, I had briefly seen Claude generate test code by using a skill which seemed to be well written. The author was a highly skilled QA engineer with a strong track record in the company. With that skill, Claude did an ok job in generating boilerplate test code, but even then it did not setup the POMs well. For example, unnecessary locators, some page functions which seem obvious to a human were not present (obvious due to named html attributes) and some other stuff I can’t remember now. I don’t know if this is because the skill was not good enough or if Claude is just not ready to write most of our tests reliably. At this point, I am not sure if Claude is very useful beyond boilerplate for test code. Have you had a different experience?

PS -
I don’t know if we really have an AI bubble. But I hope that the so called AI “bubble” bursts soon, so that we can see what is hype vs what is real. It would be so hilarious if the AI bulls end up like the self driving bulls i.e. they tell us that AI will need way more time to become viable (like self driving cars). The cars had to first become testable on public roads (which seems to be the case now) and then replace human driven cars (seems to be way ahead in the future).

6

i’d test the economics, not whether Claude can produce code. pick 10 representative scenarios across shared UI, canvas, and game-specific WebSocket behavior. compare human-written and generated versions on review time, corrections, false passes, flake rate, and maintenance after one real change.

if generation saves 20 minutes but adds 40 minutes of review and repair, it did not lower test cost. keep generated tests only where a QA can explain the oracle and an automation engineer can maintain them. the manager should have to show a lower end-to-end cost, not a bigger test count.

7

I think your concern is valid. Generating the Playwright code is probably the easy part here :slight_smile:

What I’d worry about is exactly what you mentioned - how much work AQA ends up doing to understand, review and later maintain all that generated code. If every game is different enough, you might just move the bottleneck from writing automation to reviewing/fixing automation. I’d try it first on 10-20 cases of different complexity and track how much human time they actually take end to end. Not just generation time. Then compare it with writing those tests normally. Would be interesting to see the result actually

8

These are valid questions.

Let’s start with the elephant in the room, what does your manager think you do?

Half the industry thinks manual testing is this, test cases and rote script execution, this leads to a lot of manager thinking that to and if it is the case it makes reasonable sense to give that over to machines as on the face if it its humans inefficiently simulating a machine strength activity.

You are highlighting if I have understood correctly that it is not that at all. For you like many testers it’s about “active real time critical thinking and analysis during risk detection design and investigation.”, there are many reasons humans outshine machines at this and many reasons this remains good usage of testers skills.

What you can do is try and get the manager to understand this is what has actually been happening but also build up a matrix of desired coverage, things that suit scripted automation often the already known very well things and those which suit active testing which leans to areas that discovery and investigation remains important.

So 2, you need to be careful with there strategy may not be fundamentally flawed but the premise beneath it very likely is, that matrix may help with that alongside some recent real examples.

  1. Yes - with limited coding skills you can still create and maintain basic playwright flows with the help of AI on models, architecture and good coding practices

  2. I’d choose something that does clearly fit the machine strength criteria and experiment with that, if it is successful it’s potentially a win/win. An interesting experiment would also be for things that do not lean to machine strengths, if your hypothesis is correct it may give you evidence to back your views, be open to your hypotheses being incorrect though.

Is it real? There are different camps on this, some believe the activity of testing requires active critical thinking and analysis so automated pass/fail scripts fall into the area of checking rather than testing. AI is more grey as it can experiment, can it do deep dives based on risk hypothesis is a bigger question, where oracles are really good like security there is argument it is closer to testing than checking, this is still under debate though and I am not sure its a discussion that will help you resolve your current challenge. Quoting experts tends to wind up managers more than helping.

You could always by those stakeholders pushing this a couple of testing books though.

I do not know your detailed context though but I see a lot of opportunity in agentic AI testing but not at the cost of losing the high value of real time critical thinking and analysis, there is room for both.

9

@dwaynesamuels I wonder if any team will do such an exercise to compare with ai vs without ai costs. They are too busy to do it. Even thought I got some mixed signals from Devs in my previous company, they agreed that it improves productivity for developing features (i.e. not just making scripts for odd jobs). So, I guess that this feeling might serve as a proxy for judging the value & productivity gains of AI. The only thing that can shatter this warm and fuzzy feeling is a good tester.

10

that warm feeling is exactly why the comparison has to be small enough to run inside normal work. don’t launch a study. sample the next five AI-written tests and record generation time, human review and repair, false passes, and the first maintenance change.

a tester can make the cost visible without slowing delivery. if teams refuse even that lightweight check, they are not using confidence as a proxy. they are using enthusiasm as evidence.

11

I think there’s another experiment worth trying before deciding whether the approach works: don’t measure how much Playwright Claude can generate. Measure how much useful failure evidence it produces.

Take 10-20 real historical bugs, hide them in known builds, give the agent only the intended user behaviour and see what it actually catches.

Then measure false positives, missed bugs, review time and maintenance after a few application changes.

Otherwise it’s very easy to prove “AI can create automation” while never proving that the automation improves confidence.

12

@y_k19 - I like your idea. But, I wonder how feasible it would be for most companies.

Have you tried this before? If yes, then what challenges did you face?

Was it a monolith or a microservices based app ? Was it easy to get the historical builds & deploy them? How did you ensure that the AI only gets to see the old requirements and never the latest ones or the bugs in the builds?

13

Exactly. And it is our job and chance to enhance this more and more with the team and above.

I do like the discussion and by the way - got pointed in our internal QE digest from @andreas.vogeser :slight_smile:

14

This thread mirrors something we keep running into from the product side of an AI QA tool.

@ckenst’s point about “tests passed” only being evidence about the scope we chose really resonates. The failure mode we see most often isn’t the AI writing bad code, it’s the AI (or the humans reviewing it) quietly narrowing what “passing” means until the suite says green but nobody could tell you what’s actually been verified.

The self-healing distrust makes sense too, for a related reason: healing a broken locator and healing a broken test intent are different problems. A selector can heal itself just fine while the test keeps asserting something that no longer matches what the user journey actually needs.

The suggestion above to run it on 10-20 cases and track real end-to-end human time (not just generation time) seems like the right instinct. The generation step is rarely where the cost was hiding.

@nickolas.pjescow, curious whether the review step your team has planned (AQA reviewing the MR) is scoped to “is this code correct” or “does this test still verify the right thing” — those need pretty different review habits, and conflating them is usually where the maintenance burden creeps back in.

15

@chipichipi

My two cents, maybe 3.

  1. No.

  2. Probably not, even if you prove it. At some point your manager has to trust you: your methodology, observations, numbers, approach. Most of that he can’t verify himself, so you’re back to him trusting your words.

  3. See 2.

What Id try instead? there are a lot of new concepts flying around right now. Show you know a few of them, build some PoCs and compare the results. It doesn’t fully solve the trust, but a “named” approach carries more weight than “I think”.

Personally, I’ve been using the “disposable software” idea for about a year. The point is that sometimes it’s cheaper to create from scratch than to maintain. For your situation you can look at it from a different angle:

  • Does a test, once it’s created, need to be maintained long term? Not always. You can use it during the sprint, while the data in the QA env isn’t updated and changes between builds are small. Healing the test or recreating it from scratch is cheap.

  • Is the feature ready to test? You have scenarios and use cases, so you can use a harness to write the Playwright code on the fly. If you know your app deeply or have a lot of experience, you’ll see in the report what was missed in the tests the LLM actually ran, or what you missed yourself. You can also check which code was hit during the test.

Im from backend side. And yes, many “ifs” omitted, but you have the idea.