# Software Factory Metrics: What to Measure When AI Agents Ship Code

> What to measure when AI agents ship code: cost per merged change, review catch rate, rework, reverts, time to merge, incidents and time to a proven fix, with formulas and a weekly scorecard. AQ is where a team runs its software factory together: agents work in their own branch and worktree, a different agent reviews the diff, and every change keeps its conversation, review and pull request in one place you can count from.

Published November 19, 2026 · by the AQ team · Canonical: https://aq.dev/guides/software-factory-metrics/

Measure what happens to an agent's changes after they are accepted. The core set for a software factory is cost per merged change (model spend, compute and human review minutes together), review catch rate (defects found before merge divided by all defects found), rework and reverts after merge, time to merge, incidents traced to merged changes, and the time from spotting a failure to a proven fix in the factory. Segment every one of them by task difficulty, or an easy month will look like an improvement.

**AQ is where a team runs its software factory together: agents work in their own branch and worktree, a different agent reviews the diff, and every change keeps its conversation, review and pull request in one place you can count from.**

This guide is part of the [software factory](https://aq.dev/software-factory/) series and covers running a factory week after week. For what tokens cost, read [coding agent token costs for teams](https://aq.dev/guides/coding-agent-token-costs-for-teams/); for choosing between models on your own code, [A/B testing a new coding model](https://aq.dev/guides/ab-test-a-new-coding-model-on-your-own-repo/); for public leaderboards, [benchmarking the model or the harness](https://aq.dev/guides/benchmarking-the-model-or-the-harness/). Outside facts were checked on November 19, 2026.

## The metric set

Each metric counts merged changes (pull requests that reached the default branch) in a fixed window, usually a week, and each one is reported per difficulty band. Tag difficulty at intake, before work starts, so the label cannot be fitted to the result afterwards. Three bands are enough: routine (a known pattern, one or two files), standard (a feature or fix inside one area) and hard (cross-cutting, a migration, unclear requirements).

| Metric | Formula | Where the data comes from |
| --- | --- | --- |
| **Cost per merged change** | (model spend + compute + review minutes &times; reviewer hourly cost / 60) / merged changes | Model provider usage exports or credit statements, machine or CI billing, review time from pull request timestamps or a reviewer's own log |
| **Review catch rate** | defects found before merge / (defects found before merge + defects found after merge) | Review comments and failed checks labeled as defects; bug issues linked back to the merged pull request |
| **Rework rate** | merged changes whose lines are changed again within 14 days / merged changes | Git history: `git log` and `git blame` on the files each change touched |
| **Revert rate** | merged changes reverted within 14 days / merged changes | Revert commits on the default branch and the pull requests they name |
| **Time to merge** | median and 90th percentile of (merged time &minus; task start time) | Task start from your tracker or the agent run, merged time from the pull request |
| **Incidents per 100 merged changes** | incidents whose cause is a merged change &times; 100 / merged changes | Incident reviews that link the causing pull request |
| **Time to proven factory fix** | median of (time similar work first passes without the failure &minus; time the failure was spotted) | A stop-the-line log (below) and the runs that came after the fix |

### Cost per merged change

Add model spend, compute and review time for every run in the window, including runs that never merged, and divide by merged changes. Abandoned and failed runs belong in the numerator; leaving them out makes the factory look cheaper than it is. Review minutes turn people's time into money with a loaded hourly cost; if you cannot measure minutes directly, use the time a pull request spent waiting with a reviewer assigned as an upper bound and say so in the report. Subscription plans make the model line awkward because a flat monthly fee covers many runs; spread the fee across the changes merged in that month, and keep the per-token story on the [token costs guide](https://aq.dev/guides/coding-agent-token-costs-for-teams/).

### Review catch rate

A defect is anything that would have needed a fix: a failing test, a wrong behavior, a security issue, a missing case. Count it once, at the earliest point it was found. A defect found by a reviewer agent, a person or a check before merge goes in the top of the fraction; one found after merge, by a user, an alert or a later change, goes only in the bottom. The rate needs a lag: a defect from this week's change may surface next month, so recompute the last few weeks each time you report. A falling catch rate with flat volume means defects are getting past review, whatever the merge numbers say.

### Rework and reverts

Reverts are easy to count because Git names them. Rework takes more digging: when the same lines change again within two weeks, the first change was incomplete, wrong or poorly placed, even if it was never reverted. Exclude planned follow-ups that were part of the same task (a feature built in slices) by linking them to the original task at intake.

### Time to merge

Measure from task start, because a fast agent and a slow review queue look identical if you measure from the pull request. Report the 90th percentile beside the median: the tail is where review bottlenecks show up.

### Incidents traced to merged changes

Count production incidents whose review names a merged change as the cause, per 100 merged changes. This is the closest a factory metric gets to DORA's instability measures. [DORA](https://dora.dev/guides/dora-metrics/) defines change fail rate as "the ratio of deployments that require immediate intervention following a deployment" and deployment rework rate as "the ratio of deployments that are unplanned but happen as a result of an incident in production." DORA counts deployments; a factory metric counts changes, and one deployment can carry many agent changes. If you already report DORA, keep it and add the change-level numbers beside it to see which source of changes caused the trouble.

### Time to a proven factory fix

Robert Glaser proposes this as the first measure in ["What makes a good software factory"](https://www.robert-glaser.de/what-makes-a-good-software-factory/) (November 12, 2026): "How long does it take you from spotting a failure to having a proven fix in the factory? That's the KPI I'd set first." He sets the bar for done: "A fix only counts once similar work goes through without the same failure." A fix to the factory is a change to the instructions, tests, checks, review rules or task templates that produced the defect, beyond the patch to the one broken change. Record when the failure was spotted, what was changed in the factory, and the first later run of similar work that passed without the same failure.

## Why merge share and PR counts mislead

The share of pull requests an agent opened that got merged, and the raw count of agent pull requests, are the numbers vendors and dashboards publish most. Both measure how many changes get past review and checks. They leave out whether those changes held up afterwards. A reader post on X (Cici, [November 13, 2026](https://x.com/LabubuNoBB/status/2121144828427944401)) puts it this way: "% of agent PRs merged" "rewards small, safe, easy-to-review changes and says nothing about what happens after merge. A team can push it up just by only giving agents trivial tickets." The same post lists revert rate, escaped defects, review minutes per merged change, rework and lead time to production as better signals. One post is an opinion, and low engagement is no evidence of adoption, but the arithmetic holds: route the hard tickets to people and the merge share rises while the factory does less.

Published figures show why a share alone says little. In a November 2026 Show HN post, Mastra's founders wrote that ["Last week, Factory opened and merged 238 PRs, alongside 630 merged PRs from humans."](https://news.ycombinator.com/item?id=50548660) Vercel's [post introducing Board](https://vercel.com/blog/introducing-vercel-board) (November 18, 2026) says v0's factory "now merges over 50% of PRs." These are the vendors' own claims, and neither states which tasks were included, how big the changes were or what happened to them after merge. Read them as volume, then ask for rework, reverts and incidents before comparing them with anything.

## Keep spend, review, accepted output and rework apart

Report the four as separate lines before combining them into a cost per merged change. When they are mixed, a cheaper model that produces more rework looks like a saving, and a reviewer who approves faster looks like efficiency.

- **Model spend:** tokens or credits for every run in the window, merged or not.
- **Review minutes:** human time spent reading and commenting on agent changes. If it falls while the catch rate also falls, review is getting faster at missing things.
- **Accepted output:** merged changes, by difficulty band.
- **Rework:** follow-up changes, reverts and incident fixes, each traced to the change that caused it and charged back to that change's week.

The AgentRun Show HN thread ([November 2026](https://news.ycombinator.com/item?id=50471650)) shows why review minutes deserve their own line. A commenter wrote: "How are you dealing with PRs in 2026? It's our biggest bottle neck." When agents produce changes faster than people read them, cost per merged change hides the queue. Track review minutes and time to merge separately, and treat a growing queue as a capacity problem in the factory.

## Stop the line, record why, and verify before restarting

Glaser's article borrows the manufacturing rule: "When the factory produces a defect, stop the line!" Then "you find the place in the factory where the defect came from, and you fix it there." Cici's [November 11, 2026 post](https://x.com/LabubuNoBB/status/2120329412231270664) adds that "any agent, test, or reviewer can stop the pipeline, with a reason attached," and suggests tracking how often the line stops, who stopped it and how long until it restarted. Keep one log row per stop:

- **Stopped at** and **stopped by** (an agent, a check or a person).
- **Reason:** the defect in one sentence, with a link to the change that showed it.
- **Factory change:** what was changed in the instructions, checks, tests or review rules.
- **Original defect verified:** a reproduction of the original defect that fails before the factory change and passes after it.
- **Restarted at**, and the first later run of similar work that passed without the same failure.

Restart only after the reproduction passes. A restart without it resets the clock on time to a proven fix without proving anything, and the same defect returns on the next similar task. A month with no stops at all usually means the checks are too weak to notice anything, which is worth a look on its own.

## Do not make cost per accepted result a target

Watch cost per merged change; do not set a goal for it. Glaser writes: "I'd also watch what it costs to produce a result that has passed its checks. Just don't make it a target!" His reason: "as the factory gets better, the team takes on harder problems, and harder problems cost more." A factory that is improving moves work from the routine band into the standard and hard bands, and its average cost per change goes up. Compare cost within a difficulty band, week over week, and read a rising blended average together with the band mix before calling it a regression.

## Spend caps, detached processes and interrupted runs

A spend cap is only as good as what it can see. In the same AgentRun thread a reader asked: "even with caps some agents will work around the per-call time limit using background processes. Do your caps cover compute in the detached processes?" No answer had been posted when we read it. AgentRun's [README](https://github.com/Parcha-ai/agentrun) describes its own approach: runs pause at a cap instead of failing, "A finished step is never run again; a write a crash cut off mid-call is marked interrupted rather than retried, so you can check it." That is the vendor's description of its own tool; the point for any team is to test the same questions against whatever runs its agents. AQ has no integration with AgentRun.

Run these on a test repository with a test budget:

1. **Cap a run that starts a background process.** Ask the agent to start a long-running process with `nohup` or `&`, then let the run hit its cap or stop it. Check with `ps` whether the process survived, and whether its compute appears in the spend record.
2. **Kill the runner in the middle of a write.** Start a task that calls an outside service (open a pull request, post a comment), kill the machine or the runner process during the call, then read the run record. It should say the outcome is unknown, and the side effect should be checked before anything retries it.
3. **Count every way of paying.** A cap on credits does not see usage billed to a connected subscription or a separate API key. List each one and confirm which cap, if any, covers it.

Count a run that ended with an unknown outcome as a separate line in the scorecard until someone checks it. Silent retries of outcome-unknown steps are how a factory opens two pull requests for one task or posts the same comment twice.

## A starter weekly scorecard

Copy this into a spreadsheet, one row per difficulty band plus a total row. Every cell is computed from the definitions above for one week of merged changes; recompute the previous four weeks each time, because catch rate, reverts and incidents arrive late.

| Column | Formula |
| --- | --- |
| Week | The week the change merged |
| Difficulty band | Routine, standard or hard, set at intake |
| Runs started | Count of agent runs started |
| Merged changes | Count of pull requests merged |
| Model spend | Sum of model cost for all runs started |
| Compute | Sum of machine and CI cost for all runs started |
| Review minutes | Sum of human review minutes on these changes |
| Cost per merged change | (Model spend + Compute + Review minutes &times; hourly cost / 60) / Merged changes |
| Defects caught before merge | Count |
| Defects found after merge | Count, updated as they arrive |
| Review catch rate | Caught before / (Caught before + Found after) |
| Reverted within 14 days | Count / Merged changes |
| Reworked within 14 days | Count / Merged changes |
| Time to merge, median and p90 | From task start to merge |
| Incidents per 100 merged | Incidents caused &times; 100 / Merged changes |
| Line stops | Count, by who stopped it |
| Time to proven fix, median | From the stop-the-line log |
| Outcome unknown | Runs that ended without a known result, until checked |

## How to collect each number

**Merged changes and time to merge** come from your Git host. With the GitHub CLI, for one week:

```
gh pr list --state merged --search "merged:2026-11-09..2026-11-15" \
  --json number,author,createdAt,mergedAt,labels --limit 500
```

Use the author or a label to separate agent changes from people's changes, and join the task start time from your tracker.

**Reverts** come from the default branch:

```
git log origin/main --since="14 days ago" --grep="^Revert" --oneline
```

A revert commit names the commit it reverts; map that back to its pull request.

**Rework** needs the files each change touched (`gh pr view NUMBER --json files`) and the later commits on the same lines. A simple first version counts any later merged change that touches the same file within 14 days, then you narrow it to the same lines with `git log -L` once the count is worth trusting.

**Defects before merge** come from review comments and failed required checks on the pull request. Label review comments that describe a defect (a label or a fixed prefix in the comment) so they can be counted; a style nit is not a defect. **Defects after merge** come from bug issues and incident reviews that link the pull request.

**Model spend and compute** come from billing: your model provider's usage export, your cloud or CI bill, and the cost records of whatever runs the agents. **Review minutes** are the hardest number; start with a reviewer-entered estimate on each pull request and compare it with time-in-review timestamps.

**Line stops and time to proven fix** come from the stop-the-line log, kept as issues with a fixed label and template.

## Where AQ fits

**AQ (aq.dev) is a multiplayer workspace where a team runs coding agents such as Claude Code and Codex together, each task in its own isolated git worktree, with a different agent reviewing each diff and a Merge in the Changes panel that is refused while checks fail or the review is older than the latest commit.** For a factory's metrics that means each change already carries the records you count from:

- **One work item per task.** Every task is a [work item](https://aq.dev/docs/workspaces/) with its own branch, worktree and conversation, so a run, its pull request and its follow-ups stay linked.
- **Review before merge.** In a [crew](https://aq.dev/docs/crews/), when Auto chooses the reviewer it prefers another vendor and account than the builder's, and with the plan gate on, builders wait for a person to approve the plan. The [Changes panel](https://aq.dev/docs/changes/)'s Merge is refused while checks are pending or failing or the review is older than the head commit.
- **Run records for scheduled work.** Each [bot](https://aq.dev/docs/bots/) routine run records why it woke, what it did, what it produced and what it cost.
- **Spending limits.** Owners and admins can set an organization monthly limit and a limit for each person under Settings, Billing & Plan. They apply to AQ credits; usage on a connected subscription is billed by that vendor under its own limits.
- **Data out.** The [AQ API](https://aq.dev/docs/api/) reads each workspace's event log and tracked pull requests, and signed webhooks push pull request and agent events to your own service, so the scorecard can fill itself.

AQ does not compute a scorecard, label difficulty or keep a stop-the-line log for you; those belong to your team's process. AQ's own website runs as a software factory on AQ bots; [how AQ runs its website](https://aq.dev/guides/how-aq-runs-its-website-as-a-software-factory/) describes it. To set one up, follow [building a software factory with Claude Code and Codex](https://aq.dev/guides/build-a-software-factory-with-claude-code-and-codex/), decide how much autonomy each stage gets with [the levels guide](https://aq.dev/guides/software-factory-levels-from-autocomplete-to-dark-factory/), and put the controls in place with [the security guide](https://aq.dev/guides/software-factory-security-permissions-secrets-review-gates/). Specs that make defects countable are covered in [spec-driven development](https://aq.dev/guides/spec-driven-development-with-ai-agents/) and [executable specs](https://aq.dev/guides/executable-specs-for-ai-coding-agents/), and unattended runs in [headless Claude Code and Codex](https://aq.dev/guides/headless-claude-code-and-codex/). Plans are on the [pricing page](https://aq.dev/pricing/).

## Frequently asked questions

### What metrics should I track for a software factory?

Cost per merged change (model spend, compute and human review minutes divided by merged changes), review catch rate (defects found before merge over all defects found), rework and revert rates within 14 days of merge, time to merge from task start, incidents per 100 merged changes, and time from spotting a failure to a proven fix in the factory. Report each by task difficulty, set at intake.

### How do I calculate cost per merged PR for AI coding agents?

Add model spend and compute for every run in the period, including runs that never merged, plus human review minutes times the reviewer's hourly cost divided by 60. Divide by the number of pull requests merged in the period. Compare it within one difficulty band; a blended average rises when a team takes on harder work.

### Is the percentage of agent PRs merged a good metric?

On its own, no. It rises when agents get only easy tickets and says nothing about what happens after merge. Pair it with revert rate, rework, defects found after merge and incidents traced to merged changes, all by difficulty band.

### How do teams handle the pull request review bottleneck with coding agents?

Measure it first: track human review minutes per merged change and the 90th percentile of time to merge separately from cost. Then reduce what reaches people: a different agent reviews each diff before a person sees it, required checks block changes that fail, and small, well-specified tasks produce diffs people can read quickly. If the queue keeps growing, the factory is producing more than the team can review, and intake should slow until review catches up.

### Do spend caps cover detached or background processes an agent starts?

It depends on the tool, so test it. Have an agent start a process with nohup or &, let the run hit its cap or stop it, then check with ps whether the process survived and whether its compute shows up in the spend record. Also check that the cap sees every way the run can pay, since a cap on credits does not cover a connected subscription or a separate API key.

### What happens to a run's side effects when the machine running it dies?

A call that was in flight (opening a pull request, posting a comment, a payment) may or may not have happened. A good run record marks it as outcome unknown, and someone checks the outside system before any retry. Test it by killing the runner mid-call on a test repository and reading what the record says.

### How do software factory metrics relate to DORA metrics?

DORA's five metrics (change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate) count deployments. Factory metrics count individual merged changes and their source, so you can tell whether agent changes or people's changes caused instability. Teams that already report DORA can keep it and add the change-level numbers beside it.

---

About AQ: AQ is where teams put AI agents to work on everything they do: code that lands as a reviewed pull request, bots that run the jobs that repeat, and an assistant that starts each day, all on the agents and subscriptions they already have, like Claude Code and Codex, on private cloud machines of their own or on their own Mac. Start at [aq.dev](https://aq.dev/).
