Are You Benchmarking the Model, or the Harness?
Published September 1, 2026 · by the AQ team
Every published coding agent benchmark score measures a system, not a model: the model plus the harness around it (the prompts, tools, context management, and retry loop that turn raw inference into an agent). The harness share of that number is large. As of September 2026, the same model run through different harnesses on identical tasks has been measured swinging 20 percentage points, which is bigger than most model-generation upgrades. So when a leaderboard says "Model M scores 65%," the honest reading is "this model, inside this specific harness, scored 65%," and the number does not transfer to your harness, your repository, or your workflow. This guide explains where the harness contribution comes from, what the evidence shows, and how to compare coding agents in a way that survives contact with your own codebase.
What the harness contributes to a score
A benchmark task hands the agent a repository and a goal, then checks whether tests pass at the end. Everything between those two points is harness territory:
- Context strategy. Which files the model sees, how search results are summarized, and what gets dropped or compressed when the context window fills. A model that never sees the relevant file cannot fix it.
- The tool set. Whether the model edits files through string replacement, unified diffs, or whole-file rewrites; whether it can run tests, grep, or open a browser. Aider's polyglot benchmark documents this directly: it supports several structured edit formats and configures the best one per model, because edit format alone changes pass rates.
- Loop control and recovery. How many turns the agent gets, when it is told to stop, and what happens after a failed command. Retry logic converts near-misses into passes.
- Feedback loops. Whether the harness lets the agent verify its own output. In an August 2026 five-harness experiment (same prompt, same model, published on grigio.org), the harnesses that could run headless Chromium and inspect their own output produced consistently better results. The author's summary: if the model is the brain, the harness is the eyes and the hands.
None of this is cheating; it is the work harness vendors compete on. The problem is only attribution: when a score improves, the number alone cannot tell you whether the model got smarter or the scaffolding got better.
The evidence: same model, different number
Several independent 2026 measurements quantify the harness effect:
- An August 2026 arXiv study titled "Same Model, Different Harness: Different Coding-Agent Results" ran one model (Kimi K3) through eight harnesses on the same tasks. Pass rates ranged from 68% to 88%, and the bill for a single task differed by roughly 32 times between harnesses (about $0.07 versus $2.26) with similar code quality.
- Terminal-Bench reports pairs, not models. The terminal-agent benchmark from the Laude Institute, Stanford researchers, and the open source community scores the agent and model together and requires public trajectories. Its 2.1 leaderboard is a list of agent and model pairs, and it states outright that the same model can receive different scores under different harnesses, as of September 2026.
- Standardized versus vendor scaffolds. Scale's SWE-bench Pro leaderboard runs every model through one fixed scaffold with the same prompt, precisely so the ranking reflects the model rather than scaffold tuning. Vendor-run numbers for the same models, produced inside each vendor's own tuned harness, run substantially higher. Neither number is wrong; they measure different things.
- Minimal harnesses get surprisingly far. mini-swe-agent, the SWE-bench authors' deliberately tiny harness (about 100 lines of Python, bash as the only tool, one shared prompt for every model), reports over 74% on SWE-bench Verified as of September 2026. The gap between that floor and tuned-harness numbers is a decent estimate of what scaffolding is worth on that benchmark.
- The research consensus is shifting. Harness-Bench (arXiv, 2026) evaluated harness configurations across multiple models under shared budgets (106 tasks, 5,194 trajectories), found substantial variation across pairings, and concluded that capability should be reported at the model-harness configuration level. A companion position paper, "Stop Comparing LLM Agents Without Disclosing the Harness" (May 2026), documents cases where changing the harness reverses model rankings, and proposes a disclosure standard.
Anthropic's own engineering write-up on SWE-bench says the quiet part plainly: performance on the benchmark can vary significantly based on the scaffolding, even when the underlying model is the same. That is a vendor telling you how to read its own numbers.
Why vendors report the numbers they do
Seen as system scores, vendor numbers stop looking sinister. A model vendor announcing a new model wants the best defensible number, so it runs its own tuned scaffold and documents the setup in a footnote (tool set, number of attempts, prompt additions). A standardized leaderboard wants cross-model comparability, so it freezes one harness and accepts that every model is handicapped to the same degree. A harness vendor wants to show its scaffold beats the baseline, so it holds the model fixed and varies the harness.
All three are legitimate experiments. The footnote is the actual result; the headline is marketing. Two practical consequences:
- Numbers from different reporting regimes are not comparable. A vendor-scaffold 70% and a standardized-scaffold 52% for the same model are both true. Quoting them against each other, or against a different model's number from the other regime, is comparing apples to the orchard.
- Small deltas between systems mean little. If harness substitution alone moves a score by double digits, a 2-point gap between two leaderboard entries built on different scaffolds is noise.
How to read a coding agent leaderboard
A checklist before letting any number influence a tooling decision:
| Question | Why it matters |
|---|---|
| Which harness produced this number? | If the page does not say, the number is unusable. Serious leaderboards name the agent and model as a pair. |
| Is the scaffold standardized or vendor-tuned? | Standardized scores compare models; tuned scores show a system's upper bound. Different questions, different numbers. |
| Are trajectories public? | Published run logs let anyone verify what the agent actually did. Terminal-Bench and the official SWE-bench leaderboard both require them. |
| Is cost or turn count reported? | Two harnesses can reach similar pass rates at wildly different cost per task. A score without cost hides half the result. |
| How many attempts per task? | Best-of-N sampling and generous retry budgets inflate scores relative to single-attempt runs. |
The number that matters is measured on your repo
Even a perfectly disclosed benchmark measures someone else's tasks: popular open source repositories, standard build systems, well-tested codebases. Your monorepo with the custom build wrapper and the flaky integration suite is a different distribution. The practical move is a small bake-off of complete pairings (harness plus model) on your own code: three real tasks from your backlog, each candidate pairing run from the same clean branch in its own git worktree, scored on whether it finished without rescue, whether the edits were clean, what it cost, and whether you would have merged the diff. The harness chooser guide walks that protocol step by step, and the harness directory is a reasonable shortlist to draw candidates from.
Treat the leaderboards as a filter, not a verdict: they tell you which pairings are worth the bake-off, and your repository tells you which one to adopt.
Where AQ fits
AQ is the multiplayer coding harness where engineering teams run AI coding agents like Claude Code and Codex together: shared live terminals, a code editor, and app previews, in your own cloud. In this guide's terms, AQ sits a level above the harnesses being benchmarked: it is a harness of harnesses, the place where the real CLIs (Claude Code, Codex, Cursor Agent, Kimi, Grok, or plain shells) run side by side rather than a scaffold that replaces them.
That makes it a natural bench for exactly the comparison this guide recommends. Each workspace gets its own isolated git worktree, so the same task can run in several harnesses at once from the same clean branch without the runs trampling each other. Every session runs in a persistent tmux session on your team's VM and streams live to the browser, so teammates can open the same workspace and watch the same run, and a bake-off survives someone closing a laptop. Each engineer signs into the CLIs with their own Claude or OpenAI account, and AQ never marks up model usage, so the costs you observe are the provider's real prices, not a platform's bundled rate.
Pricing is two plans: Free is a personal sandbox for one person (AQ creates a private machine in an isolated network, nothing to install, no time limit), and Team is $50 per user per month early access (standard $200, billed monthly), covering VMs you connect from your own cloud or a dedicated always-on AQ-managed VM, with your rate locked for your first 12 months.
Plainly: AQ will not tell you which harness scores highest on SWE-bench, and does not try to. It gives your team a place to run the pairings that matter on the repository that matters, watch them work, and decide from evidence instead of a leaderboard screenshot.
Frequently asked questions
Why do vendors report higher SWE-bench scores than standardized leaderboards?
Because they run their own tuned scaffolds. A vendor number shows the upper bound of that model inside that vendor's harness, with its choice of tools, prompts, and retry budget. Standardized leaderboards like Scale's SWE-bench Pro freeze one scaffold across every model to make the model comparison clean. Both are honest measurements of different things, and as of September 2026 the gap between the two regimes for the same model is routinely double digits.
How much can the harness change a coding benchmark score?
By more than most model upgrades. An August 2026 arXiv study ran the same model through eight harnesses on identical tasks and measured pass rates from 68% to 88%, a 20-point spread, with per-task costs differing by roughly 32 times. Terminal-Bench states directly that the same model receives different scores under different harnesses, which is why its leaderboard reports agent and model pairs rather than bare models.
What is a coding agent harness, exactly?
The software layer that turns a raw model into a working agent: the system prompts, the tool set (file editing, shell, search), context management, loop control, and error recovery. Claude Code, Codex CLI, OpenCode, and Aider are all harnesses. The coding harness glossary page covers the term in depth, and the harness directory maps the current landscape.
Can I compare two models using scores from different leaderboards?
No. A score is only comparable to another score produced under the same harness, task set, and attempt budget. Comparing a vendor-scaffold number for one model against a standardized-scaffold number for another compares two experimental setups, not two models. If the leaderboard does not disclose its harness, treat the number as unverifiable.
What is the fairest way to compare coding agents for my team?
Run complete pairings (harness plus model) on your own repository. Pick three real backlog tasks, run each candidate from the same clean branch in an isolated git worktree, and score finish rate, edit quality, cost, and whether you would merge the diff. Use public leaderboards only to shortlist candidates. The choosing a coding harness guide describes the full protocol.