How to A/B Test a New Coding Model on Your Own Repo

Published September 27, 2026 · by the AQ team

To find out whether a new coding model is better on your code, run the same real task twice: once on the model you use today (the control) and once on the new one (the candidate), each in its own isolated git worktree, then diff the two branches and score both with your repository's own test suite. The whole experiment takes about ten minutes of setup, works in any agent CLI that lets you pick a model (Claude Code, Codex, Cursor CLI, and OpenCode all do, as of September 2026), and answers the only question that matters on launch day: not "is it better on a leaderboard" but "is it better on my repo, my conventions, my build."

The question recurs because launches recur. In September 2026 alone, OpenAI began rolling out GPT-6 Astra on September 3 and Xiaomi released the MiMo-V2.6 family with MIT-licensed weights on September 22. Each launch week the same thread appears in every harness community: is it actually better on real work? A repeatable experiment is the honest answer.

Why public benchmarks cannot answer this

A benchmark score is a measurement of the model on someone else's code, under someone else's harness, on tasks that were public before the model trained. Three gaps follow:

• Contamination and age. Widely used suites like SWE-bench are built from public GitHub issues, and 2026 evaluation literature keeps finding that older, widely circulated problems leak into training data and inflate scores. A high score can measure memory, not skill.
• Domain mismatch. Public benchmarks over-represent popular Python and JavaScript projects. If your value is in a legacy monolith, internal APIs, or an unusual stack, the benchmark distribution simply is not your distribution.
• The harness is in the loop. An agent's result is the product of the model and the scaffolding around it: system prompts, tool definitions, retry logic, context management. Vendor scores are measured under the vendor's harness, which is rarely the one you run. Are you benchmarking the model, or the harness? unpacks this, and the harness tax quantifies how much the wrapper alone can move results.

None of this makes benchmarks useless: they are a fine coarse filter. But the decision "switch my daily driver" deserves a measurement on your own repository, and that measurement is cheap to run.

The ten minute playbook

Step 1: pick one real task. Choose a task from your actual backlog: a bug with a known reproduction, a small feature with clear acceptance criteria, or a refactor your team has been putting off. Write the prompt once, in a file, so both models receive byte-identical instructions. Do not write a toy task; the differences between frontier models show up on the messy, multi-file work, not on fizzbuzz.

Step 2: one isolated worktree per model. A git worktree gives each run its own working directory and branch while sharing one object store, so the two agents can run at the same time without touching each other's files:

git fetch origin
git worktree add ../trial-control -b trial/control origin/main
git worktree add ../trial-candidate -b trial/candidate origin/main

Install dependencies in each (or copy a warm cache). Isolation is the whole trick: same starting commit, same prompt, the model is the only variable. Git worktrees for AI coding agents covers the mechanics and the cleanup.

Step 3: run the control and the candidate. Start your harness in each worktree, pinned to the right model (exact flags per harness below). Run them in parallel; separate worktrees make that safe. Let each agent run to completion without coaching. If you would normally intervene, intervene identically in both or not at all: a steered run and an unsteered run are not comparable.

Step 4: score with the repo's own tests. Run your full test suite, typecheck, and linter in both worktrees. These are the ground truth you already trust, and they are exactly what public benchmarks cannot use.

cd ../trial-control && npm test && npm run typecheck
cd ../trial-candidate && npm test && npm run typecheck

Step 5: diff the branches. Read both diffs the way you would review two pull requests for the same ticket:

git diff origin/main..trial/control --stat
git diff origin/main..trial/candidate --stat
git diff trial/control..trial/candidate

Passing tests is the floor, not the verdict. Look at diff size (did one model touch 40 files to fix a 2 file bug), whether it followed the conventions in your instructions file, whether it invented dependencies, and whether it deleted or weakened tests to get to green.

Step 6: keep the artifacts. Record the prompt, both model identifiers with exact version strings, the date, wall-clock time, token spend, and your verdict in a short note in the repo. The branches themselves are the evidence; the note is what makes run three comparable to run one.

Switching models in each harness, as of September 2026

All four major CLIs let you pin the model per session, which is what a controlled trial needs. Verified against each vendor's documentation in September 2026:

Details worth knowing. In Claude Code, aliases like sonnet, opus, and haiku resolve to the latest model in each family, so for a reproducible trial prefer the full model name, and note that /model <name> also saves your default (open the picker and press s to switch for the session only). Codex CLI accepts the model per invocation with -m and stores a default in config.toml; the GPT-6 family (Astra, Sol, Luna) began rolling out on September 3, 2026, and /model also exposes reasoning effort, which you should hold constant across both runs. Cursor's CLI made agent the primary entrypoint (cursor-agent remains an alias) and added the agent models command, a --list-models flag, and the /models slash command in its January 8, 2026 release. OpenCode sets the model as provider/model in opencode.json, takes --model on the command line, and can override the model per named agent.

Re-run it two weeks later

A launch-day result is a sample of the model and the vendor's serving stack on that day, and serving stacks change after launch. The clearest public example: Anthropic's September 2025 postmortem of three infrastructure bugs that intermittently degraded Claude's output quality between August and early September 2025 (at the worst hour, on August 31, 16 percent of Sonnet 4 requests were affected), with fixes landing between September 2 and September 12. Anyone who benchmarked in that window measured the bugs, not the model. Launch weeks also bring capacity strain and rapid inference-stack iteration, in both directions.

So schedule a second identical run about two weeks after launch, from the same base commit, with the same prompt file. If the verdict holds twice, switch with confidence. If it flips, you have learned something a one-shot test never shows: run the trial again before you trust either result. Teams that run several agents anyway can fold this into normal work by giving the candidate model a fraction of real tasks in parallel agent sessions and comparing pull request outcomes over a week.

Where AQ fits

AQ is the multiplayer coding harness where engineering teams run AI coding agents like Claude Code and Codex together: shared live terminals, a code editor, and app previews, in your own cloud. The playbook above is exactly the shape of an AQ workspace: every workspace gets its own isolated git worktree on a fresh branch (ai/{id}-{slug}) with dependencies installed automatically, so a control run and a candidate run are two workspaces side by side instead of two directories you wire up by hand. The agents inside are the real CLIs (Claude Code, Codex, Cursor Agent, Kimi, Grok, or plain shells) running in persistent tmux sessions on your team's VM, so a long candidate run survives a closed laptop, and teammates can open both workspaces and watch the two models work the same task live. You sign into each CLI with your own Claude or OpenAI account (AQ never marks up usage on your own subscriptions), so the trial costs whatever your existing subscription costs. When both runs finish, each workspace tracks the pull requests its agent opened, which makes the two-week re-run easy to compare against the first.

Pricing is two plans: Free is a personal sandbox for one person (AQ creates a private machine in an isolated network, nothing to install, no time limit), and Team is $50 per user per month early access pricing (standard $200, billed monthly), covering VMs you connect from your own cloud or a dedicated always-on AQ-managed VM, with the rate locked for your first 12 months. If you already script worktrees comfortably, the manual playbook above is enough; AQ earns its place when you want the trials isolated, persistent, and visible to the whole team by default.