The harness tax: does your coding harness even matter?
Published September 20, 2026 · by the AQ team
The harness tax is the share of a coding agent's cost that pays for the harness itself: the system prompt, tool schemas, and scaffolding the CLI sends on every request, rather than the model's work on your task. In September 2026 it stopped being a vibe and became a measured number: benchmarks that hold the model fixed and swap only the harness find success rates moving by a couple of percentage points while cost roughly doubles. So the honest answer to "does the harness even matter" is: a great deal for what you pay and how each engineer likes to work, much less than most teams assume for whether the task gets solved. Which reframes the question underneath, "which harness should we standardize on": for most teams, you should not.
What the harness tax is
A coding harness is the software around the model: the CLI or app that builds the context, defines the tools, loops on the model's output, and executes what it decides. All of that machinery rides inside the same token budget as your task. Portkey's April 13, 2026 essay that popularized the term defines the tax as "every token your agent spends on itself before it spends a single token on your task," and measured it: given the same task, Pi sent about 2,600 input tokens, Codex about 15,000, and Claude Code about 27,000, a roughly 10x spread that is pure harness before any work happens. Over a 40-turn session they measured Claude Code consuming about 1.12 million input tokens, roughly half of it harness overhead.
On flat monthly subscriptions the tax does not show up as dollars; it shows up as usage limits arriving sooner. And it compounds at the org level: FutureAGI's 2026 treatment sketches an illustrative 30-engineer team paying $40,000 a month and getting nothing for around $14,000 of it, once idle seats, never-read context, a frontier model renaming variables, and duplicate sessions are added up.
What the 2026 measurements actually show
Three independent efforts put numbers on the question this year, and they agree more than they disagree.
The HarnessTax study (harnesstax.github.io, published September 16, 2026 by researchers at UC Berkeley and Arena, and on the Hacker News front page the same day) ran seven models through three harnesses (Claude Code, Codex, and Pi) on SWE-bench Lite and Terminal-Bench 2.0, holding everything else in the evaluation stack fixed. The headline: the average harness effect on success rate stays within about 2 percentage points on SWE-bench Lite and about 5 on Terminal-Bench 2.0, while cost diverges hard: Claude Code cost about 2.0x Pi and 1.6x Codex on SWE-bench Lite, and about 1.5x Pi on Terminal-Bench 2.0. On the strongest Claude model tested, success was 97.8 percent in Claude Code, 96.7 in Codex, and 96.7 in Pi, at $1.33 per attempt in Claude Code versus $0.67 in Pi: twice the price for 1.1 points. One mechanism is visible in the data: Claude Code's mean initial context is more than 10x Pi's, from longer instructions and larger tool schemas.
HarnessRouter's same-task benchmark (as of September 2026) ran one real task across eight harness and model configurations, five runs each, and measured a spread of roughly 475x in cost per task and more than 3x in p95 latency. The decomposition matters more than the scary top number: most of the 475x is model-tier economics, and on the same model the harness multiplies cost by up to 2.1x. Same shape as the HarnessTax result: the model sets the price class, the harness multiplies it, and neither moves success much among mainstream choices.
Artificial Analysis launched its Coding Agent Index in May 2026, scoring full stacks (specific model plus harness pairs, not models alone) with token usage and cost per task as first-class columns: the industry's main independent benchmarker deciding that a model score without a named harness is incomplete.
| Source (as of September 2026) | Setup | Effect on success | Effect on cost |
|---|---|---|---|
| HarnessTax (UC Berkeley and Arena, Sep 16, 2026) | 7 models x 3 harnesses, fixed evaluation stack | Within about 2 points (SWE-bench Lite), about 5 (Terminal-Bench 2.0) | Up to 2.0x between harnesses on the same model |
| HarnessRouter same-task run | 8 harness and model configs, 5 runs each | Varies by task | About 475x overall; up to 2.1x from the harness alone on one model |
| Portkey (April 13, 2026) | Same task, overhead tokens compared | Not measured | About 2,600 vs 15,000 vs 27,000 input tokens of overhead per request |
The counterpoint: when the harness is the binding constraint
Before concluding the harness is a commodity, read the other side. A 2026 preprint, "Stop Comparing LLM Agents Without Disclosing the Harness," argues that on long-horizon tasks the harness is often a stronger determinant of performance than the model it wraps: in its analysis, holding the model fixed and changing only the harness raised Terminal-Bench 2 pass at 1 from 69.7 to 77.0 percent, and it cites benchmark monitoring reporting up to 15 percentage points of scaffold-only variation on SWE-bench Verified.
Both findings can be true, because they measure different distances. The convergence results compare mature, mainstream CLIs around frontier models: the harnesses your engineers would actually shortlist. The divergence results measure the gap between a well-built harness and a bare or mistuned one, plus the misattribution when a leaderboard treats a harness gain as a model gain (covered in are you benchmarking the model or the harness). Practically: any serious harness gets the task done at similar rates, and the surviving differences are cost, latency, and fit with how each engineer works.
So should your team standardize on one harness?
Mostly no, and the benchmarks above are the reason. Standardization is what you do when a choice has a quality consequence, and this one mostly does not: among mainstream harnesses on frontier models you are choosing a price multiplier and a cockpit, not an outcome. Forcing thirty engineers into one CLI buys a tidier invoice and a morale tax, not better code.
What deserves standardizing is the layer underneath, where the team-level failure modes live: where sessions execute and how long they survive, how parallel work is isolated so agents do not collide, whose credentials the agents hold, and who can see a session while it runs and after. None of those are harness properties. They belong to the environment the harnesses run in, which is the argument of the harness of harnesses pattern: treat the CLI as a per-developer preference and make the environment the shared, governed thing.
The visibility half is not hypothetical: a July 2026 LeadDev analysis of 25,264 agent-generated pull requests across 2,361 popular GitHub repositories found that in 79 percent of agentic PRs the same developer both reviewed and modified the agent's contribution, and only about one in eight workflows involved multiple humans. No harness choice fixes that, because it is not a harness property. For the personal pick, how to choose a coding harness walks the questions that actually differ, and the harness directory maps the field.
How to measure your own harness tax
Every number above was measured on someone else's tasks. Yours is a one-afternoon experiment:
- Pick three real tasks from your last sprint: a small fix, a medium feature, a gnarly debug.
- Run each through two harnesses on the same model, from the same clean branch, to a finished PR or a clear stop.
- Capture cost and time per session: the CLIs report token usage and cost, API dashboards give authoritative totals, and wall clock plus your interventions are worth recording by hand.
- Repeat each pairing two or three times before believing a difference; single agent runs are noisy.
- Set defaults, not mandates. If one harness reliably does your small fixes at half the spend, make it the default for that task class and leave the choice with the engineer.
If the measurement says your spend problem is real, the levers that actually move it are model routing, fan-out discipline, and context hygiene, which we cover in what coding agents cost to run for a team.
Where AQ fits
AQ is the multiplayer coding harness where engineering teams run AI coding agents like Claude Code and Codex together: shared live terminals, a code editor, and app previews, in your own cloud.
AQ's position on the harness question is the one the data supports: you do not have to pick. Agents run as the real CLIs your engineers already chose (Claude Code, Codex, Cursor Agent, Kimi, Grok, or plain shells) in persistent tmux sessions on the team's VM, streamed live to the browser. Each engineer signs into the CLIs with their own Claude or OpenAI account, and AQ never marks up usage on your own subscriptions, so the tax you pay goes to your own subscription at its own price and the experiment above stays clean. Every workspace gets its own isolated git worktree, which makes the comparison trivial: open two workspaces, give two harnesses the same task, and watch both live. Teammates open the same workspace and see the same session as it runs, so the result becomes shared team knowledge instead of one person's anecdote, and sessions survive a closed laptop and run in parallel without colliding.
To be precise about what AQ does not do: it does not shrink any harness's token overhead; that is between the harness and your subscription. What it removes is the reason teams reach for standardization, by making the environment (execution, isolation, credentials, visibility) the shared, governed layer while the harness stays a personal choice. The Free plan is a personal sandbox for one person: AQ creates a private machine in an isolated network, nothing to install, no time limit. Team is $50 per user per month early access (standard $200, billed monthly), covering VMs you connect from your own cloud or a dedicated always-on AQ-managed VM, with your rate locked for your first 12 months.
Frequently asked questions
What is the harness tax?
The harness tax is the share of a coding agent's cost that pays for the harness itself rather than your task: the system prompt, tool definitions, and scaffolding the CLI sends with every request. Portkey, who popularized the term in April 2026, defines it as every token the agent spends on itself before it spends one on your task, and measured the same task costing about 2,600 input tokens of overhead in Pi versus about 27,000 in Claude Code. On flat subscriptions the tax shows up as usage limits arriving sooner rather than as dollars.
Does the coding harness affect success rates?
Less than most people expect, among mainstream harnesses on frontier models. The HarnessTax study (September 16, 2026, UC Berkeley and Arena) ran seven models through Claude Code, Codex, and Pi and found the average harness effect on success within about 2 percentage points on SWE-bench Lite and about 5 on Terminal-Bench 2.0, while cost differed by up to 2x on the same model. The caveat: academic work also shows a badly built or bare harness can cost 7 points or more on the same benchmarks, so the convergence holds between serious harnesses, not between a harness and none.
Which coding agent harness is cheapest to run?
In the September 2026 HarnessTax measurements, Pi was the cheapest of the three harnesses tested and Claude Code the most expensive: about 2x Pi's cost on SWE-bench Lite for the same model, driven partly by an initial context more than 10x larger. But per-task variation is large (HarnessRouter measured up to 2.1x from the harness alone on one model, with most cost spread coming from model choice), so the reliable answer is to run two or three of your own tasks through two harnesses on the same model and compare the session costs the CLIs report.
Should our team standardize on one coding agent harness?
For most teams, no. The 2026 measurements show harness choice barely moves success rates among mainstream CLIs, so a mandate buys consistency of spend, not better code, and it takes away a tool preference engineers care about. Standardize the layer underneath instead: where agent sessions execute, how parallel work is isolated, whose credentials agents hold, and who can see sessions while they run. Set per-task-class defaults from your own measurements and leave the final pick with each engineer.
How do I measure the harness tax on my own workload?
Take three real tasks from your last sprint, run each through two harnesses on the same model from the same clean branch, and capture per-session token usage and cost (the CLIs report both; API dashboards are authoritative), wall clock, and how many times you had to intervene. Repeat each pairing two or three times before trusting a difference. An afternoon of this beats any public leaderboard, because published numbers are measured on other people's tasks.