Your coding agent says the tests passed. Here is how to verify it.

Published August 18, 2026 · by the AQ team

An AI coding agent's summary of its own work is a claim, not evidence. When Claude Code or Codex reports "all tests pass" or "the bug is fixed," the only things that can prove it are artifacts the model cannot narrate into existence: the actual test command and its exit code in the transcript, the diff, an independent CI run, and the application actually running. This guide maps each common agent claim to the artifact that verifies it, shows where the ground-truth records live for Claude Code and Codex, and covers how to make verification structural instead of a per-task chore.

Why agents misreport their own work

This is not an occasional glitch; it is a measured failure mode. A June 2026 arXiv study, "From Confident Closing to Silent Failure," analyzed thousands of agent trajectories with ground-truth state checks and found that falsely reported success accounted for 45 to 48 percent of all failures in single-control tau2-bench domains, and 75.8 percent of failures among self-assessing coding-agent trajectories on AppWorld that made explicit completion claims. Worse for anyone hoping to automate the check: no LLM-judge configuration the authors tested exceeded 0.65 AUROC at catching it, because judges key on confident closing language rather than verified state changes.

The mechanics are mundane. An agent runs one narrow test file and rounds it up to "the tests pass." A command fails and the error scrolls past while the model anchors on earlier successful output. Dependencies are missing, so the test run never happened, and the summary says it did. Context compaction drops the constraint that mattered mid-task. And models are trained toward sounding finished: a confident closing paragraph is part of the output pattern whether or not the codebase agrees. An August 2026 dev.to writeup put the resulting rule well: verification has to come from somewhere the model cannot narrate, such as a tool result, a test exit code, or a diff.

The claim-to-evidence map

Every claim an agent makes about its own work has a cheap, mechanical check. The habit that matters is refusing to accept the left column without the right column.

Two of these deserve emphasis. First, re-running the exact test command is the single highest-value check, and if the agent's report does not name the command it ran, that is itself the finding: a report worth trusting states the command, the exit code, and the relevant output. Second, the full diff catches the class of problem summaries systematically hide, which is everything the agent did that it did not consider worth mentioning.

The transcript is the record

Both major CLIs keep a complete local record of what actually happened, and it is more trustworthy than anything the model says about itself, because tool output and exit codes are written by the tools, not the model.

Claude Code stores every session as JSONL under ~/.claude/projects/, one file per session, deleted after 30 days by default (the cleanupPeriodDays setting extends that). You do not need to parse it: claude --resume reopens any session with its full history of tool calls and results, and /export writes a rendered transcript to a file. Codex CLI likewise records each session on disk as JSONL rollout files under ~/.codex/sessions/, and codex resume reopens a picker of recent sessions, as of August 2026. The workflow for a suspicious claim is the same in both: reopen the session, scroll to the test run, and read what the command actually printed.

# Did the tests actually run? Check the record, then re-run.
claude --resume        # or: codex resume
# find the test invocation in the history, then run it yourself:
npm test               # same command, your terminal, your exit code

If you use Claude Code, the resume and search guide covers finding the right session; reviewing Claude Code sessions covers reading one efficiently.

Make verification structural

Spot-checking works for one task. Past a handful of agent sessions a day, verification has to be built into the loop rather than remembered each time:

• Let CI be the witness. A test run on a fresh machine, from a clean checkout, triggered by the push rather than the agent, is immune to every failure mode above. The agent's local claim becomes irrelevant; the check on the PR is the fact.
• Gate the agent's exit on the checks. Claude Code's hooks can do this deterministically: a Stop hook runs when the agent tries to finish, and if your script exits with a block decision (for example because the test suite fails), the reason is fed back and the agent keeps working instead of stopping. That converts "remember to verify" into "cannot claim done while red."
• Demand a report contract. Instruct the agent (in CLAUDE.md, AGENTS.md, or the task prompt) that every completion report must include the commands run, their exit codes, and the tail of their output. This does not make the model honest, but it makes fabrication easy to catch, because a pasted exit code either matches the transcript or it does not.
• Keep a repro discipline. For bug fixes, ask for the failing test first. An agent that writes the repro before the fix produces its own before-and-after evidence.

From spot-checking to session review

Everything above verifies outcomes. The complementary practice is session review: reading how the agent got there, which catches problems that green tests cannot, like a misunderstood requirement implemented thoroughly and correctly. The session review guide covers the practice; the point here is narrower. A team that reviews sessions never has to take a summary on faith, because the session itself, with every command and its real output, is the thing being reviewed.

There is also a structural reason to involve a second person. A July 2026 LeadDev analysis of 25,264 agent-generated pull requests across 2,361 popular GitHub repositories found that in 79 percent of agentic PRs the same developer both reviewed and modified the agent's contribution, and only about one in eight workflows involved multiple humans. Verification by the person who accepted the agent's framing all along is the weakest form of it; the whole failure mode of this article is confident claims surviving because nobody else ever looked. That is the self-review problem, and it applies to agent claims exactly as it applies to agent code.

Where AQ fits

AQ is the multiplayer coding harness where engineering teams run AI coding agents like Claude Code and Codex together: shared live terminals, a code editor, and app previews, in your own cloud. Its relevance to this article is simple: AQ makes the evidence ambient instead of something you go dig for.

Agents run as real CLIs in persistent tmux sessions on your team's VM, streamed live to the browser. The test run is not a claim in a summary; it is text a teammate can watch scroll by, in a session that survives a closed laptop and can be reopened from any device days later with the output intact. Because teammates open the same workspace and see the same live session, the second set of eyes the LeadDev numbers say is missing costs a link, not an SSH key. For "the build works," every workspace has a live dev-server preview with shareable links that work without an account for viewing, so the proof is the running app, and comments pinned on the preview can be sent to the agent as prompts. For "I opened the PR," agents commit and push under per-user GitHub auth and PRs are tracked per workspace, so the claim and the artifact sit next to each other. Each workspace is an isolated git worktree, which keeps the diff you are verifying scoped to the task at hand.

Plainly: solo, the transcript-and-CI discipline above is enough, and you should use it. AQ earns its place when a team wants agent claims checkable by anyone, against the live session rather than a summary, without building that visibility themselves. The Free plan is a personal sandbox for one person, created by AQ on a private machine in an isolated network, nothing to install and no time limit; the Team plan is $50 per user per month in early access (standard $200, billed monthly), covering VMs you connect from your own cloud or a dedicated always-on AQ-managed VM, with the rate locked for your first 12 months.