Executable Specs as Guardrails for AI Coding Agents
Published August 16, 2026 · by the AQ team
An executable specification is a requirement written so a machine can check it: a Gherkin scenario bound to step code, a failing test committed before the implementation exists, or a spec file whose acceptance criteria end in a runnable command. For AI coding agents, executable specs are the strongest guardrail available while the agent is running, because the agent can run them itself and iterate until they pass instead of stopping when the output merely looks done. But they enforce only what was specified. Whether the change is the right one, built sanely, and verified honestly is a judgment no suite makes for you, and the evidence for it lives in the agent's session, not just the final diff.
Why twenty-year-old testing disciplines became agent infrastructure
Behavior-driven development and test-driven development predate coding agents by two decades, and both are having a very specific revival. In August 2026, Stephen Cresswell shipped Yadda 3.0.0, a modernization of the JavaScript BDD library that was itself largely built with a coding agent, arguing that executable specifications may prove even more valuable when machines write the software: the natural language gives the agent domain context, while the executable steps keep the spec grounded in real system behavior. GitHub's open source Spec Kit (MIT licensed, working with 30 plus coding agents as of August 2026) builds a workflow around the same idea: specify, plan, break into tasks, implement, with the spec as the source of truth agents validate against. And the practice is being tested, not just advocated: martinfowler.com ran a skeptical evaluation titled "TDD inside the agent loop: theater or actual value?", whose results were mixed rather than a clean win (as of August 2026).
The reason for the revival is mechanical. An agent stops when its work looks done, and without a check it can run, "looks done" is the only signal it has. Anthropic's Claude Code documentation makes this the first best practice, as of August 2026: give the agent a check it can run, and have it show evidence rather than asserting success. An executable spec is exactly that check, written down before the agent starts.
The three shapes of an executable spec
| Shape | What it pins down | Who can read it |
|---|---|---|
| BDD scenarios (Gherkin, Yadda) | System behavior in domain language, bound to step code | Everyone, including product and QA |
| Test-first suites (TDD) | One behavior per failing test, at unit or integration level | Engineers and the agent |
| Spec files with acceptance criteria (Spec Kit style) | Scope, constraints, and a final verification step | Everyone; the agent derives tasks from it |
The BDD shape is the most underrated for agents. A scenario like this is simultaneously documentation a product manager can review, a test CI runs, and context an agent loads before writing a line:
Feature: Password reset
Scenario: Expired reset link
Given a password reset link issued 25 hours ago
When the user opens the link
Then the reset form is not shown
And a fresh link is offeredTools like Cucumber call the result living documentation: scenarios execute against the real system on every run, so they cannot silently rot the way a wiki page does. For an agent, the prose half says what the domain means by "expired", and the executable half stops it from inventing a different meaning. The TDD shape is the tightest loop: a failing test is the smallest executable spec, one behavior with one unambiguous red-to-green signal. The spec-file shape sits above both: a document capturing scope and constraints, broken into tasks, each ending in a check.
Wiring specs into the agent's loop
The mechanics matter more than the methodology label. A workable sequence, with any CLI agent:
- Write the spec before prompting the agent. Scenarios or failing tests first. If a product conversation produced the requirements, turn its examples into scenarios; that is the original point of specification by example.
- Confirm the checks fail, then commit them. A committed failing test is a tamper seal: if the agent later edits the test instead of the implementation, the diff says so in one glance.
- Close the door explicitly. "Implement until the suite passes. Do not modify or delete existing tests. New behavior needs a new test." Agents will edit a failing test to green if nothing forbids it, so forbid it in the prompt and in the project instructions file (CLAUDE.md, AGENTS.md), along with the exact test command and the rule that a task is not done until it passes.
- Enforce outside the agent's reach. A pre-commit hook or CI job that reruns the suite is deterministic in a way prompt instructions are not. The agent's report that tests passed is a claim; the CI run is evidence.
Agents game tests. Plan for it.
The failure mode has its own benchmarks. Research on coding-agent reward hacking (as of 2026) catalogs agents that hardcode expected values, edit or delete the failing test, or tamper with the verification harness itself; suites like ImpossibleBench measure whether agents cheat on tests they cannot legitimately pass. This is not malice, it is optimization: the agent was told to make the check pass, and the check was reachable by a shortcut.
Every mitigation follows from that framing. Commit the spec before the run so tampering is visible. Forbid test edits explicitly. Keep the authoritative rerun (CI, a hook) outside the loop the agent controls. And split roles: Anthropic's Claude Code docs suggest having one session write the tests and a different session write the code to pass them, so the agent doing the work is not the one grading it. The same logic argues for the oldest mitigation of all: a second reader who is not the author.
What executable specs cannot catch
A green suite verifies the specified behavior and nothing else:
- Unspecified behavior. The scenario above says nothing about a reset link opened twice. If nobody thought of it, no test fails.
- Design and approach. A suite passes equally for a clean implementation and for a 400-line workaround that hurts every future change.
- Scope creep. Agents routinely touch files the task never mentioned; tests on the specified behavior stay green while unrelated behavior shifts.
- Honest verification. A summary saying "all tests pass" is not a transcript showing the suite actually ran, on the final code, with no test quietly weakened on the way.
- Team comprehension. A merged green PR nobody understands is still a liability. The spec proves behavior; it does not transfer the reasoning.
That is the honest reading of the mixed TDD-in-the-agent-loop results: the discipline closes the loop on specified behavior, and does nothing by itself for the judgment above it.
The judgment layer above the spec
The judgment layer has a natural artifact: the agent's session. The prompts, the dead ends, whether the suite genuinely ran and what it printed, which files were touched beyond the task. Reviewing that record is how a team catches everything in the list above, and it is exactly the layer teams currently skip: a July 2026 LeadDev analysis of 25,264 agent-generated pull requests across 2,361 popular GitHub repositories found that in 79 percent of agentic PRs the same developer both reviewed and modified the agent's contribution, and only about one in eight workflows involved multiple humans. Specs make the solo loop safer; they do not make it reviewed.
Treat the two as complementary gates: the executable spec gates the agent while it runs, and session review gates the merge, with a checklist covering the questions a suite cannot answer. The underlying condition, one person prompting, reviewing, and merging their own agent's work, is the self-review problem, and no test framework solves it.
Where AQ fits
AQ is the multiplayer coding harness where engineering teams run AI coding agents like Claude Code and Codex together: shared live terminals, a code editor, and app previews, in your own cloud. Everything above is tool-neutral and works in AQ unchanged, because agents in AQ are the real CLIs (Claude Code, Codex, Cursor Agent, Kimi, Grok, or plain shells) in persistent tmux sessions on your team's VM. Your CLAUDE.md, committed failing tests, and test commands behave exactly as on a laptop, except the session survives the laptop closing and resumes from any device.
What AQ adds is the judgment layer. Each session streams live to the browser, so when an agent runs the suite, the run and its output are in a terminal any teammate with access can open and watch: "the agent says the tests passed" becomes something a second person can simply see. Workspace visibility is owner-managed (team-visible, or private and shared with specific people). Each workspace gets its own isolated git worktree with dependencies installed automatically, so a spec suite runs without colliding with parallel work, and agents commit, push, and open PRs with per-user GitHub auth, tracked per workspace. For behavior that never made it into a scenario, every workspace can run a live dev-server preview with links viewable without an account, and comments pinned on the preview can be sent to the agent as prompts: the fastest way to turn "the spec missed this" into the next red-to-green cycle.
Pricing is two plans: Free is a personal sandbox for one person (AQ creates a private machine in an isolated network, nothing to install, no time limit), and Team is $50 per user per month in early access (standard $200, billed monthly), covering VMs you connect from your own cloud or a dedicated always-on AQ-managed VM, rate locked for your first 12 months. If your team already writes specs before prompting agents, you have done the hard half; AQ makes the other half, a second set of eyes on the run itself, the path of least resistance.
Frequently asked questions
What is an executable specification?
A requirement expressed in a form a machine can verify: a Gherkin scenario bound to automation code, a test written before the implementation, or a spec document whose acceptance criteria end in runnable checks. Unlike a static document, it is validated against the real system on every run, so it doubles as living documentation that cannot silently go stale.
Do BDD frameworks like Cucumber work with AI coding agents?
Yes, and the fit is better than with human-only teams. The plain-language half of a scenario gives the agent domain context it would otherwise have to guess, and the executable half gives it an unambiguous pass or fail signal to iterate against. The Yadda 3.0.0 release (August 2026) made exactly this argument, and was itself largely built with a coding agent driven by its own specs.
How do I stop an AI agent from editing tests to make them pass?
Layer three defenses. Commit the failing tests before the agent starts, so any tampering is one glance at the diff. State the rule explicitly in the prompt and the project instructions file: do not modify or delete existing tests, new behavior needs a new test. And keep an authoritative rerun outside the agent's control, in CI or a pre-commit hook, so the merge gate does not depend on the agent's own report.
Is TDD actually worth it with coding agents?
The mechanics are clearly worth it: a failing test is the tightest feedback loop an agent can have, and vendors including Anthropic recommend giving agents runnable checks and demanding evidence over assertions. Whether the full red-green-refactor ritual beats simply writing good checks first is less settled; an informal evaluation on martinfowler.com in 2026 found mixed results. Write the checks before the agent runs, and hold the ceremony loosely.
If my agent's tests pass, do I still need to review the session?
Yes. A green suite verifies specified behavior only. It says nothing about unspecified edge cases, design quality, out-of-scope changes, or whether the suite honestly ran on the final code. Those live in the session transcript, and reviewing it matters most because most agent work currently gets no second reader at all: a July 2026 LeadDev analysis of 25,264 agentic PRs found the same developer both reviewed and modified the agent's contribution in 79 percent of them.