What AI Code Review Can and Cannot Catch
Published August 23, 2026 · by the AQ team
AI code review is genuinely good at what static analysis of source text can reach: mechanical bugs, known vulnerability patterns, cross-file inconsistencies, and subtle logic slips that tired humans skim past. What it cannot reliably catch, as of August 2026, are the failure modes that live outside the text of the code: behavior that only appears when the code runs, code that is correct but never invoked in the shipped configuration, changes that do the wrong thing correctly because the intent was never written down, and bugs the reviewing model is predisposed to miss because it shares assumptions with the model that wrote the code. An AI approval means the change reads as correct, not that it is correct, and the difference is the set of checks that still belong to you.
What AI review reliably catches
Credit first, because the catch rate is real, and the better tools long ago stopped being diff-only. As of August 2026, per each vendor's documentation: CodeRabbit clones the repository into a sandbox, builds a code graph, and runs dozens of linters and static-analysis tools; Greptile indexes the whole codebase and follows references across files when evaluating a change; Cursor's Bugbot reads how the changed code interacts with the rest of the codebase and targets logic and security bugs rather than style; GitHub's Copilot code review scans the diff plus surrounding context and suggests concrete fixes.
On that footing, AI review is strong at null handling, boundary slips, resource leaks, race-prone patterns, injection-shaped vulnerabilities, error paths that swallow failures, and inconsistencies with the conventions elsewhere in the repo. It reviews every line with the same attention, at 2am, on every PR, and a team that skips this layer in 2026 is leaving cheap defect detection on the table.
The Coldcard lesson: the same tool found it and missed it
The sharpest recent illustration of both edges is the July 2026 Coldcard incident. A bug introduced in March 2021 caused certain Coldcard hardware wallet firmware versions (4.0.1 through 4.1.9) to skip the dedicated hardware random-number chip during seed generation: a library checked whether a build setting existed, not whether it was switched on, and key generation quietly fell through to a weak software substitute. The flaw sat in public, open-source firmware for over five years before attackers began draining weakened wallets in late July 2026, with press coverage putting losses above 100 million dollars within about a week.
Two details in Coinkite's own advisory matter here. First, because the source code was always public, the company assumes someone used a frontier AI model to review old firmware and found the issue. Second, a few weeks earlier Coinkite had run one of the best available AI models over the same code looking for security issues, and it did not find this bug or anything serious.
Read those together and you have the honest state of the art: AI review can surface a five-year-old flaw that human review, audits, and a security-conscious community all missed, and the same class of tool, pointed at the same code by the defenders, can come back clean. AI review findings are evidence. AI review silence is not.
Failure mode 1: nothing ever runs
Pre-merge review, human or AI, reasons about code as text. It does not execute the change against real data, real configuration, or real third-party services, so behavior that only exists at runtime is structurally out of reach: the query that is correct but takes ninety seconds on the production table, the API that returns a shape the docs did not promise, the migration that locks a hot table. The Coldcard bug is the canonical shape: every line involved read plausibly, and the failure lived in the interaction between a build setting's value and a library default, visible only in what the device actually did.
Failure mode 2: correct code that never runs
A related blind spot: the diff adds a well-written function, a handler, a validation, and the reviewer verifies it is well written. Whether the surrounding system ever calls it is a different question. Feature flags that default off, registration steps that live in configuration, environment variables that differ between staging and production: all of these can leave a reviewed, approved, merged change dormant in the shipped product. Codebase-aware reviewers can flag an obviously unreferenced function, but reachability in the deployed configuration is not in the diff, and an agent asked to add a check will happily add it somewhere nothing executes. The only convincing evidence that code runs is watching it run.
Failure mode 3: the reviewer inherits the author's framing
AI reviewers read the PR description, and it sways them. A March 2026 study, Measuring and Exploiting Confirmation Bias in LLM-Assisted Security Code Review, found that biasing an LLM reviewer toward believing the code is bug-free cut vulnerability detection by 16 to 93 percent depending on model and setup, with false negatives rising sharply while false positives barely moved. Simply redacting the pull request description recovered most of the missed detections. This matters doubly for agent-written PRs, where the description was generated by the same process that wrote the code. A reviewer that starts from the author's summary is reviewing the intention, not the change; we cover this in how to verify AI coding agent work.
Failure mode 4: correlated blind spots
When the model family that wrote the code also reviews it, errors correlate: the wrong assumption that produced the bug is available to rationalize it during review, and an edge case the generator did not consider is often one the reviewer does not consider either, because both were shaped by similar training. The practical mitigation is diversity: a different model for review than for generation, plus static tools that do not share LLM biases. But the deepest version, an agent whose tests, summary, and self-review all descend from the same misunderstanding, is not fixable inside the AI layer. It needs an independent observer. That is the self-review problem, and it is organizational, not technical.
What catches what the reviewer cannot
Each structural gap has a matching check, and none of them is another pass of static review:
| Blind spot | What actually catches it |
|---|---|
| Runtime behavior | Run the change: tests that execute the path, and a live preview of the running app |
| Code that never executes | Trace the change from a real entry point in the running product, in the shipped configuration |
| Author framing bias | Review the work, not the summary: the session transcript and the diff, before the description |
| Correlated blind spots | A second, independent reviewer: a different model, and above it a second human |
| Wrong intent, correct code | A human who knows what the product needed, looking at the running result |
The last two rows are where teams are currently thinnest. A July 2026 LeadDev analysis of 25,264 agent-generated pull requests across 2,361 popular GitHub repositories found that in 79 percent of agentic PRs the same developer both reviewed and modified the agent's contribution, and only about one in eight workflows involved multiple humans. In most agentic work today, the checks AI review cannot do rest on one person's attention. If your AI reviewer is drowning that person in comments, start with cutting AI review noise; for how a human should review an agent's PR, see reviewing pull requests from AI agents.
Where AQ fits
AQ is the multiplayer coding harness where engineering teams run AI coding agents like Claude Code and Codex together: shared live terminals, a code editor, and app previews, in your own cloud. AQ is not an AI code review tool and does not replace one; it provides the evidence layer for the checks static review cannot do.
Agents run as real CLIs (Claude Code, Codex, Cursor Agent, and others) in persistent tmux sessions on your team's VM, streamed live to the browser, so reviewing the work can mean watching what the agent actually did: which tests ran and what they printed, not what the summary claims. Teammates open the same workspace and see the same live session, which is how a second human gets into the loop without a handoff ceremony. Every workspace gets a live dev-server preview with a shareable link that works without an account for viewing, so "does this change actually run, wired into the real app" is a click, not a checkout; comments pinned on the preview can be sent to the agent as prompts, and outside reviewers can comment through a review link after a quick email sign-in. Each workspace is an isolated git worktree on its own branch, with the agent's PRs tracked per workspace, so the diff, the session that produced it, and the running result stay attached to each other.
Plainly: keep your AI reviewer and let it catch everything text analysis can reach. The failure modes in this guide are caught by running the change and by a second pair of human eyes on the session and the preview. AQ's job is making both cheap enough that they actually happen.
Frequently asked questions
Is AI code review still worth using if it misses these things?
Yes. It catches mechanical bugs, known vulnerability patterns, and cross-file inconsistencies with perfect stamina, on every PR, in minutes. The mistake is not using it; the mistake is treating its approval as the end of review. Treat it as the first reviewer whose silence proves nothing, and reserve human attention for runtime behavior, wiring, and intent.
Can AI code review replace human review?
Not as of August 2026, and the vendors say so themselves: GitHub's documentation for Copilot code review states it may not identify all problems, especially in large or complex changes, and should be supplemented with careful human review. The structural reasons are that pre-merge review never executes the change and cannot know unstated product intent. What AI review does replace is the mechanical share of human review effort.
Why do AI reviewers approve code that turns out to be broken?
Three structural reasons: the review is static, so behavior that only exists at runtime is invisible; the reviewer reads the PR description, and a March 2026 study found that framing a change as safe cut LLM vulnerability detection by 16 to 93 percent; and when the reviewing model shares training and assumptions with the model that wrote the code, they tend to miss the same things. None of these is fixed by running the same review twice.
What kinds of bugs can AI code review not catch reliably?
Anything whose evidence is outside the source text: performance against production data shapes, third-party APIs behaving differently than documented, configuration and feature-flag wiring that leaves correct code unreachable, migrations that are safe in staging and dangerous on real tables, and changes that are internally correct but solve the wrong problem. The matching checks are executing the change, tracing it from a real entry point, and a human who knows the intent.
How should I verify a change that AI review has approved?
Run it and watch it. Execute the tests that cover the changed path and read their actual output, exercise the change in a running instance of the app from a real entry point, and read the diff before the PR description so the author's framing does not anchor you. For agent-written changes, the session transcript is the record of what really happened. On a team, the strongest cheap check is a second human looking at the running result.