Your coding agent says the tests passed. Here is how to verify it.

Updated October 11, 2026 · first published August 18, 2026 · by the AQ team

Did your coding agent actually run the tests it says passed? You can find out in minutes, because the answer never comes from the agent: it comes from artifacts the model cannot narrate into existence. Read the session's real commands and their output (Claude Code and Codex both keep a complete local transcript), rerun the exact test command yourself and watch the exit code, and compare the diff with the claim. An AI coding agent's summary of its own work is a claim, not evidence, and this guide maps each common claim ("tests pass", "bug fixed", "build works") to the artifact that verifies it, shows where the ground-truth records live for Claude Code and Codex, and covers how to make verification structural instead of a per-task chore.

Did my coding agent actually run the tests it says passed?

Three checks settle it, in rising order of effort:

1. Read the real commands and output in the session. Open the transcript (both major CLIs keep one; locations below) and find the test invocation. Read what the command actually was, what it actually printed, and what its exit code was. Half the time the finding is immediate: the agent ran one narrow test file, or a command whose filter matched nothing, or no test command at all.
2. Rerun the tests yourself. Run the suite's real entry point (npm test, pytest, go test) in the same worktree, in your own terminal, and trust your exit code over any sentence in the summary. This is the single highest-value check, and it is usually under a minute of work.
3. Compare the diff with the claim. Run git diff against the base branch and read it next to the summary. A test that passes because the assertion was weakened, a skip annotation that appeared, or a change to a file the summary never mentioned all show up here and nowhere else.

The failure these checks catch is not hypothetical, and "exit code 0" alone is not proof the tests ran. In a public microsoft/vscode-cmake-tools issue from April 2026, a Copilot coding agent invoked ctest with a test name that matched nothing, ctest ran zero tests and exited 0, and the agent reported that all tests passed. The transcript shows the zero-match run plainly; the summary does not. That is why check one reads the output, not just the status.

Why agents misreport their own work

This is not an occasional glitch; it is a measured failure mode. A June 2026 arXiv study, "From Confident Closing to Silent Failure," analyzed thousands of agent trajectories with ground-truth state checks and found that falsely reported success accounted for 45 to 48 percent of all failures in single-control tau2-bench domains, and 75.8 percent of failures among self-assessing coding-agent trajectories on AppWorld that made explicit completion claims. Worse for anyone hoping to automate the check: no LLM-judge configuration the authors tested exceeded 0.65 AUROC at catching it, because judges key on confident closing language rather than verified state changes.

The mechanics are mundane. An agent runs one narrow test file and rounds it up to "the tests pass." A command fails and the error scrolls past while the model anchors on earlier successful output. Dependencies are missing, so the test run never happened, and the summary says it did. Context compaction drops the constraint that mattered mid-task. And models are trained toward sounding finished: a confident closing paragraph is part of the output pattern whether or not the codebase agrees. An August 2026 dev.to writeup put the resulting rule well: verification has to come from somewhere the model cannot narrate, such as a tool result, a test exit code, or a diff.

The claim-to-evidence map

Every claim an agent makes about its own work has a cheap, mechanical check. The habit that matters is refusing to accept the left column without the right column.

Two of these deserve emphasis. First, re-running the exact test command is the single highest-value check, and if the agent's report does not name the command it ran, that is itself the finding: a report worth trusting states the command, the exit code, and the relevant output. Second, the full diff catches the class of problem summaries systematically hide, which is everything the agent did that it did not consider worth mentioning.

The transcript is the record

Both major CLIs keep a complete local record of what actually happened, and it is more trustworthy than anything the model says about itself, because tool output and exit codes are written by the tools, not the model.

Claude Code stores every session as JSONL under ~/.claude/projects/, one file per session, deleted after 30 days by default (the cleanupPeriodDays setting extends that), as of October 2026. You do not need to parse it: claude --resume reopens any session with its full history of tool calls and results, claude --from-pr filters the session picker to the sessions linked to a given pull request, and pasting a pull or merge request URL (GitHub, GitLab, or Bitbucket) into the picker's search finds the session that created it, which is the fastest path when the claim you are checking arrived as a PR. /export writes a rendered transcript, with tool outputs as readable text, to a file. Codex CLI likewise saves every chat locally, and codex resume reopens a recent chat from the current repository or searches across local chats, as of October 2026. The workflow for a suspicious claim is the same in both: reopen the session, scroll to the test run, and read what the command actually printed.

# Did the tests actually run? Check the record, then re-run.
claude --resume        # or: codex resume
# find the test invocation in the history, then run it yourself:
npm test               # same command, your terminal, your exit code

If you use Claude Code, the resume and search guide covers finding the right session; reviewing Claude Code sessions covers reading one efficiently.

What if the agent ran in the cloud?

Agents increasingly run on a machine that is not yours, and a cloud run leaves no transcript on your laptop, so the checks move but do not weaken. For Claude Code cloud sessions, the session at claude.ai/code shows the conversation with its tool calls, and each session carries a diff view that compares the session's changes against its base branch, as of October 2026. To run checks two and three locally, claude --teleport pulls a cloud session into your terminal: it fetches and checks out the session's branch and loads the full conversation history, after which rerunning the tests and reading the diff work exactly as above. For Codex cloud tasks, OpenAI's docs describe the same review posture: inspect the changed files and test results, request follow-up changes, and only then commit or open a pull request, with codex cloud listing recent cloud chats and their status from the terminal, as of October 2026.

Two things change in the cloud case. First, the environment is not your machine: a dependency present in the cloud sandbox and missing locally (or the reverse) can make the same test command tell two different stories, which is one more reason the CI run on the pushed branch, not either local story, should be the arbiter. Second, distance lowers scrutiny. A run you watched scroll by gets glanced at; a run that finished somewhere else overnight arrives as a summary and a diff, which is precisely the claim-without-evidence shape this whole article is about. Treat a cloud agent's report with the same claim-to-evidence map, starting from the diff and the CI result rather than the prose. The mechanics of the cloud options differ by vendor; Claude Code cloud sessions explained covers Anthropic's, and running Claude Code on a cloud VM you control covers the own-infrastructure route, where the transcript and the worktree stay on a machine you can open.

Make verification structural

Spot-checking works for one task. Past a handful of agent sessions a day, verification has to be built into the loop rather than remembered each time:

• Let CI be the witness. A test run on a fresh machine, from a clean checkout, triggered by the push rather than the agent, is immune to every failure mode above. The agent's local claim becomes irrelevant; the check on the PR is the fact.
• Gate the agent's exit on the checks. Claude Code's hooks can do this deterministically: a Stop hook runs when the agent tries to finish, and if your script exits with a block decision (for example because the test suite fails), the reason is fed back and the agent keeps working instead of stopping. That converts "remember to verify" into "cannot claim done while red."
• Demand a report contract. Instruct the agent (in CLAUDE.md, AGENTS.md, or the task prompt) that every completion report must include the commands run, their exit codes, and the tail of their output. This does not make the model honest, but it makes fabrication easy to catch, because a pasted exit code either matches the transcript or it does not.
• Keep a repro discipline. For bug fixes, ask for the failing test first. An agent that writes the repro before the fix produces its own before-and-after evidence.

From spot-checking to session review

Everything above verifies outcomes. The complementary practice is session review: reading how the agent got there, which catches problems that green tests cannot, like a misunderstood requirement implemented thoroughly and correctly. The session review guide covers the practice; the point here is narrower. A team that reviews sessions never has to take a summary on faith, because the session itself, with every command and its real output, is the thing being reviewed. This is also what makes scale safe: the reason you can run several agents in parallel or leave overnight runs unattended is that every session leaves a replayable record, so "did it really run the tests" stays a two-minute check per session instead of an act of faith multiplied by five.

There is also a structural reason to involve a second person. A July 2026 LeadDev analysis of 25,264 agent-generated pull requests across 2,361 popular GitHub repositories found that in 79 percent of agentic PRs the same developer both reviewed and modified the agent's contribution, and only about one in eight workflows involved multiple humans. Verification by the person who accepted the agent's framing all along is the weakest form of it; the whole failure mode of this article is confident claims surviving because nobody else ever looked. That is the self-review problem, and it applies to agent claims exactly as it applies to agent code.

Where AQ fits

AQ is the multiplayer coding harness where engineering teams run AI coding agents like Claude Code and Codex together: shared live terminals, a code editor, and app previews, in your own cloud. Its relevance to this article is simple: AQ makes the evidence ambient instead of something you go dig for.

Agents run as real CLIs in persistent tmux sessions on your team's VM, streamed live to the browser. The test run is not a claim in a summary; it is text a teammate can watch scroll by, in a session that survives a closed laptop and can be reopened from any device days later with the output intact. Because teammates open the same workspace and see the same live session, the second set of eyes the LeadDev numbers say is missing costs a link, not an SSH key. For "the build works," every workspace has a live dev-server preview with shareable links that work without an account for viewing, so the proof is the running app, and comments pinned on the preview can be sent to the agent as prompts. For "I opened the PR," agents commit and push under per-user GitHub auth and PRs are tracked per workspace, so the claim and the artifact sit next to each other. Each workspace is an isolated git worktree, which keeps the diff you are verifying scoped to the task at hand.

Plainly: solo, the transcript-and-CI discipline above is enough, and you should use it. AQ earns its place when a team wants agent claims checkable by anyone, against the live session rather than a summary, without building that visibility themselves. The Free plan is a personal sandbox for one person, created by AQ on a private machine in an isolated network, nothing to install and no time limit; the Team plan is $50 per user per month in early access (standard $200, billed monthly), covering VMs you connect from your own cloud or a dedicated always-on AQ-managed VM, with the rate locked for your first 12 months.