As AI powered testing keeps evolving , I kind of wonder what the community thinks about all this.
Traditional automation is great for checking the scripted, predefined scenarios , but actual people don’t really walk the same straight line all the time. AI agents are supposed to go in a more lively way , exploring the app, adapting to UI changes, and generally doing that “human-like” interaction vibe.
Would it mainly uncover:
Usability and UX snags?
Odd customer journeys?
Accessibility gaps?
Business logic defects?
Performance slowdowns and bottlenecks?
Or maybe something else entirely
I’m genuinely curious about your experiences, what you saw in practice, and your opinions. Lets talk
i’d look first for stateful bugs that sit between the scripted paths: a role changes while a tab stays open, back/forward restores stale UI, a control appears only after a particular sequence, or timing changes which request wins.
the agent matters less than the handoff. if it finds one, save the starting state, exact action, first contradiction, and browser context as one replayable test artifact engineering can inspect. otherwise “the agent found it” just creates another reproduction job.
the real test is not whether it acts human-like. it is whether another person can verify what it did.
Let’s clarify, real users generally do not test your product they use your product but yes using the product can reveal bugs.
A professional tester does something different, they build risk models around the value and intent of the system, design risk hypothesis and experiments to investigate those risks. If a big exists their experiments are designed optimally to find it.
This is an important different, if an agent is able to simulate a certain type of user what sort of bugs do you think will come up. It’s much more challenging if at all possible to simulate a professional tester, we just have far too much tacit knowledge right now that agents have not been trained on or in some cases the capacity to learn.
What you add though is observability to your flow, you may not have thought of a test case but with an agent crawling your app things will come up as its able to observe everything and it gives an extra layer to your scripted test cases. There is an argument it will find shallow easy to find issues but to be fair it can be guided for more than that.
Oracles and heuristics are important to an agent, they extend it beyond a real user. Where the oracles are strong the results will be better. A security agent for example will often outperform a tester who has not studied security risk deeply as it has access to a lot of oracles both on what the risks are but also how to investigate them. Accessibility is mid level on this w3c has a very strong guide and the agent can leverage tools like axe fairly easily.
Other risks though there is often not enough oracles or heuristics available particularly on new risks that go with innovative products, for these perhaps an interactive agent is better suited or maybe you could spend time building skills for the agent based on your own risk profiles and context.
It adds a layer of real time observability, it benefits from good oracles but does not replace professional testing level if your testers are good at what they do, they can be paired though.
Oh I should add loads of automation suites don’t do what I am suggesting either even though they are designed by professionals, so on that basic question it can add a layer of observability to your current coverage, again though even on scripted testing an automator would at this point still likely outperform an agent due to the knowledge and experience they have.
