We have started to use the AI functionality within our QA tool (spending too many credits) and it’s been great for creating test cases and test runs. We still do the manual checks and some have to be rewritten, but overall it saves us time for sure. But what tasks does AI still do badly?
I’m not sure about the idea of badly, most birds are good at flying but I’m not going to suggest I fly badly as that would only be the case if I was given the job of a bird for some reason.
What you can do is consider activities it has strengths in alongside tasks not really appropriate to hand over to AI.
Would you give activities that require critical thinking to a machine for example? It’s real time application is the primary strength of hands on testing so activities where it is required real time might not be a great match for AI.
Similarly it does not have the values of having lived a human life, no experience of nature, nurture, growth, sadness, joy, empathy - so again activities that benefit from those likely not suitable.
Then we have accountability, its a machine, it has almost no sense of consequences, do not falsely believe you can give it accountability.
It does though have a lot of strengths, sometimes start with a basic matrix of activities and give scores for human or machine strengths, there will be some it’s capable of but can result in loss of an indirect value, team knowledge creation and retention is one many flag for concern as activities pass over to agents.
Be aware the some teams compete with machines and already try and do machine strength activities, a bit like asking me to fly well, these teams can likely get AI to do everything without much loss.
reproducing stateful browser bugs from an incomplete report. ai is useful once the starting state is explicit, but it still tends to guess when the failure depends on what happened earlier: a role change, stale tab, back/forward navigation, timing, or which request won.
a good test is to ask it for the first action where expected and observed behavior diverged, plus the browser state at that moment. if it only rewrites the symptom or jumps to a likely cause, it has summarized the bug rather than investigated it.
Anything you can’t ground. Test cases work for you because you still check them manually, that’s your grounding. Without deterministic validation you can’t be sure about result, so you just pray, or trust your own history of which AI results were good and which were bad
Pick for the handoff rather than the test run. Most QA tooling is good at finding and tracking; the expensive step is getting a finding to a developer with enough context that they do not have to reproduce it themselves.
(post deleted by author)
the pray had me rolling lol 
100% this. AI is great at describing the symptom but bad at finding where things first went wrong. That’s why in https://www.testbrick.com/ I only let AI write the test once. After that it’s plain Playwright, and when a run fails it shows you the exact first step that went wrong.