I’m a developer, and I’m trying to calibrate whether something I’m seeing is now standard practice or just particular to my team.
Recently our QA started running AI over our pull requests. Not only to decide “safe to merge / needs work” — the AI also writes concrete change suggestions: rename this, extract that, handle this edge case, restructure that function.
Honestly, I’m open to it. Some suggestions are things I’d have wanted from review anyway, and it’s an extra set of eyes that never gets tired. A reviewer that catches the boring stuff is welcome.
But a chunk of the suggestions are “textbook good” and impractical here — correct in the abstract, wrong for our constraints, our performance budget, or the way this part of the code actually works. I’ve started declining those and explaining why. Which occasionally feels strange: I’m pushing back not on a person, but on a model my QA colleague is trusting.
So I’m curious how common this is, and how
Is QA-run AI code review — with concretenot just a gate decision — already normal onyour team?
When the advice is “good but impractical,” how does it get filtered? Does the dev just decline case by case, or does QA triage the AI’s output before it re
I don’t think the answer is "AI shouldn’t serested in how teams keep those suggestionsas useful signal, instead of a queue of well-meaning noise the dev has to relitigate.
i’d treat the model like an untrusted reviewer, not qa’s verdict.
before a suggestion reaches the developer, someone should be able to explain the actual risk, the constraint it protects, and why it fits this codebase. otherwise the developer becomes the filter for a queue of plausible-sounding noise.
qa using ai here can be useful, but i’d only escalate findings a human is willing to understand and defend.
This “QA code review” should be for informational purposes only. Not a blocker to the CI pipeline.
As you said, quickly go through the “QA code review” comments, fix them whenever you think it makes sense, and forget about the rest (no action needed)
This is interesting I have just been discussing this sort of thing and whether its part of the newish Engineer type role that some testers are being encouraged to transition to.
The roles whilst very varied at the moment can for some be viewed as looking at both the software and the system that the product evolves within from a quality engineering perspective, this can broaden it from testing. Two examples I used earlier to question this were influencing how effective code reviews were alongside helping to define merge and branch controls, clearly not testing remit but areas where quality engineering can be applied.
The risk is exactly what you are seeing is they can start creating a system of work that they have no expertise in, it will rightly look like they are trying to do a developers job if they approach it from the point of view they own it rather than being in an assistance QE capacity helping developers establish the right system for their needs.
The review process going to agents is likely to happen sooner or later, it carries risk your “good but impractical” is common on initial usage of agents, I used an accessibility one and it started off similarly creating noise and slowing developers down but I’d expect these to improve fairly quickly.
I’d suggest this introduction of AI into the review system needs expert input alongside the quality expertise the QE might bring, it will take a few rounds to get this right but the intent is likely a good one.
I’m a tester rather than a QA or QE and its this specific reason, not enough expertise in a specific area of the development system that makes me cautious about this role change.
Curious too, as I don’t yet use any AI tools with code myself.
It’s surely just an advanced linting tool - if you have been using a linter already, then it’s not going to show you much more we would hope. I’d love if someone can give us an idea how much cleverer than a linter this is? And how to tell it to ignore code? Can it do that? Being unable to do that is the main reason very few people use linters because they start to litter legacy code with ignore decorations.
Getting a QA to suggest fixes is a bit daft, especially if you are letting them use tools they are not integrating into the product lifecycle and process. Sounds like a “shift-right” thing to give tools to QA which might break things late and not rather get coders to use the tools to “shift-left”, and remove the need to run the tools a second time.
Thanks all — this landed somewhere useful. The recurring line (Dwayne put it sharpest) is: treat the model as an untrusted reviewer, and only escalate a finding a human can actually defend. Advisory, not a gate.
What crystallized for me is a sharper distinction underneath that: the valuable QA output is a quality signal, not an improvement suggestion.
An example. We were migrating a large table into monthly shards. My query walked the shards in logical order and returned once it had a full page. The AI reviewer wanted it parallelized for speed. I declined.
At release, devops forgot to index the shard tables — slow queries, alerts. Because the reads were sequential, it stayed a warning. Fanning out N× concurrent queries against unindexed shards, as suggested, would have turned the same missing-index mistake into a real incident.
Contrast: in testing, QA noticed the algorithm was hitting shards it didn’t need to for some users. That made me fix it.
The parallelism advice was an improvement suggestion — locally reasonable, blind to a constraint it couldn’t see. QA’s observation was a quality signal — real observed behavior, decision left to me.
To Conrad’s linter question: a smarter linter just produces better suggestions. The output I trust is the signal — what the system actually did — because that stays true in contexts the adviser can’t see.
Improvement suggestions tell you what to do; quality signals tell you what’s true.
This brings a related question, given the tech industry and elsewhere is embracing AI. As devs and in software development, should not AI (optionally) be involved in the development and PR code review process?
For the OP’s case, seems like QA introduced the AI code review into the PR. But that should also have been a dev initiated effort into the SDLC with respect to PRs and CI/CD on the code aspects. In my organization, we’ve used copilot AI and its integration with github, making use of copilot for pull requests. In some cases, automatic trigger of copilot upon pull request creation. In other cases, optional ability to request copilot review of PRs whenever desired by the user. We also use copilot within IDE like VS Code, pycharm, IntelliJ, to help with writing code, unit tests, and you could also ask it to do code reviews there outside of pull requests.
From personal experience:
like @dwaynesamuels suggests, treat the feedback as coming from another (peer) reviewer. It could be untrusted, and feedback is always up for debate/assessment as well.
some AI + git PR integration pipelines can be tuned, such as if using the github + copilot integration. In the copilot comments/suggestions, you can rate them as thumbs up or down, which goes into improving the copilot AI model. I recall it used to have a feature where you could also provide contextual text feedback to your (negative) ratings as well, but I think github may have taken that away now. You could also respond to the AI comments/suggestions - that I’m unaware whether github pulls that into its feedback loop for improving the copilot AI or not, maybe they do, which is why they took away the negative rating feedback feature? If one is using a different AI than copilot or not using github for PRs, then you may not have as tight an integration for usefulness of AI in PRs, so it kind of depends on the tooling and integrations.
Responding to AI comments/suggestions is useful to keep a history of decisions on why you accepted or declined the suggestion, for tracking and reference, for yourself, and others who may look at the PR’s history/details. The only annoyance here is when AI repeats suggestions and whether you want to repeat your responses to those iterations or not.
Also from experience and somewhat related to this:
QA used AI to provide feedback to me as dev in a Jira/bug report, not directly in a pull request in this case. The feedback was useful in this one case as it suggested specific fixes in code, saving me time in finding the affected areas of code to fix.
But how often that happens depends on how good the AI is and how well the user (e.g. QA) did with the prompting against the AI for the given scenario (bug/code analysis, debugging, etc.). You don’t always get such insightful feedback down to the code level for fixes, but sometimes you do.
I think the problem here is that we are using linters already at more than one level, the AI is just another linter, surely unless it has broader context than just the code? Why treat this tool differently? The way the AI tool includes an upvote downvote feature is a broken abstraction,An abstraction of the same way as we turn off warnings or disable then in certain files for linting, the linting tool generations even know about each other’s markers. What is fundamentally differing?
Perhaps one way to look at it is the AI is an advanced, next level linter, that can be customized/trained, though not directly by you in the case of the upvote/downvote feature (along with any optional comments supplied).
Traditional linters, I believe, you can’t train or tweak. Your options are to simply make minor customizations like diabling warnings/errors being flagged or skip certain files that you mentioned. The other option is you modify the source code of the linter to customize it to your liking if you have the source code to it.
AI in terms of the models/LLM has capabilities to be trained based on (user) feedback loop. While the upvote downvote feature may be a broken abstraction, it’s still a feedback loop to the AI for improvement as metrics/KPIs for what is working well and what is not. So the (AI) developers can work towards tuning the AI to be better at its weaknesses.
As a user of AI, you get the improvements on the AI in future usage, and you might not need “upgrade” the software if it’s implemented as part of the service or existing model (rather than need to select the next model update/upgrade).
Keep in mind the upvote/downvote isn’t simply just a vote, I assume the system that github has may attach contextual metadata that the general user doesn’t think about. Like it may attach along references to the specific code being commented on, the proposed suggestion for correcting/improving issue. If not the raw data itself, some form of anonymized version of it. So the feedback system can take those into account for what worked or went wrong with that specific scenario being “linted”.
I imagine, that always 2 things happen, or fail, in all AI applications. 1 the AI has not got the deep context, it often simply has not got enough tokens to know not to try make suggestions about a piece of code that we don’t care about improving. OR 2. That it merely hallucinates. It’s having to parse a lot of context which linters often don’t build large enough data trees for, so it can just miss something?
In our team we put special emphasis on “human in the loop” when making use of AI in our work. From your post, it seems like the QA on your team is just throwing the whole bunch of changes towards AI and then throwing back the whole slop from it back to you.
I’d say the QA should themselves be aware of your coding style and either fine tune their agent to respect that style, or leave out potentially hallucinated AI feedback. What I think you need is like a SKILL.MD that acts as a reviewer knowing fully well how your code works.