How cheap can an autonomous builder be?

ClaroCal is built, in part, by a fleet. Overnight, six agent seats, three on Claude Code and three on Codex, work in their own git worktrees: a lead coordinates, builders and a QA seat take tickets, and each brand reviews the other’s pull requests before CI and an automated review let them merge. It runs on two consumer subscription plans, and it works well enough that the question stopped being “can agents ship fixes” and became “what does each fix cost”.

The night that got urgent, the Codex plan’s weekly meter went from 0 to 99% in about an hour, and after a manual reset the fleet burned most of a second window in three hours. Nothing was broken. Each Codex seat ran a frontier model at high reasoning effort, re-sending roughly 140K tokens of context on every call. Volume, not a bug.

So: how cheap can an autonomous builder be before you pay for it anyway, in review rounds and escaped bugs? The prize is a model cheap enough to keep the swarm running around the clock.

Method: re-ship what already shipped

Our merge history is the cleanest ground truth we have. Every merged fix comes with the issue that asked for it, the commit it branched from, the diff that passed review, and the tests that diff added: a question, a starting point, a marking scheme, and a model answer.

The harness takes seven merged fixes, small to hard, and makes every candidate sit the same exam:

Starting point. Every attempt begins at the parent of the merge commit, in a scratch clone with push disabled and GitHub credentials stripped. The model gets the original issue text as its only prompt, inside a short brief: implement this test-first, run focused tests and the typecheck, do not commit or push. Each case’s prompt is saved to a file, so any model from any vendor can be asked the identical question later.

# one scratch clone per (case, model); nothing in it can reach a PR
git clone --shared --no-checkout "$REPO" "$DIR"
git -C "$DIR" remote set-url --push origin DISABLED
git -C "$DIR" checkout --detach "$MERGE_COMMIT^1"
ln -s "$REPO/node_modules" "$DIR/node_modules"
export GH_TOKEN= GITHUB_TOKEN= GIT_TERMINAL_PROMPT=0 CI=1

Signal 1: hidden tests. After the model stops, the shipped PR’s own test files are checked out over the attempt and run. Two guards sit in front of that number. A control check runs the same tests at the merge commit; if they do not all pass there, the environment is broken and the run aborts. A discrimination check runs them on the untouched base; if they already pass there, they cannot tell attempts apart and the judge scores that case alone.

# control: the shipped tests must pass where they shipped
git -C "$MERGE_DIR" checkout "$MERGE_COMMIT" -- $TEST_FILES && run_tests  # all green, or abort
# discrimination: they must FAIL on the untouched base, or they prove nothing
git -C "$BASE_DIR"  checkout "$MERGE_COMMIT" -- $TEST_FILES && run_tests  # some red, or judge-only
# the attempt: the same files dropped over whatever the model wrote
git -C "$ATTEMPT"   checkout "$MERGE_COMMIT" -- $TEST_FILES && run_tests

Signal 2: new type errors. The typecheck runs on the attempt and on the base, errors are normalised to “file: message” with positions stripped, and only errors absent from the base count.

Signal 3: a blind judge. A Claude Opus judge sees the issue, the shipped diff, and every attempt’s diff, shuffled and labelled A, B, C. Per attempt it returns solves (0 to 3, where 3 means at least as good as what shipped), regression risk (0 to 3), expected review rounds, and a yes or no on “would you merge this after at most one small fix”. It is told that hidden tests assert names the shipped implementation chose.

Judge builder attempts at one issue against the change that actually shipped.
Hidden tests are the shipped PR's own tests; a correct fix with different
names can fail them, so weigh the diff itself. JSON only.
<issue>…</issue>
<shipped>```diff …```</shipped>
### Attempt A (hidden tests 24/24, typecheck pass) ```diff …```
### Attempt B (hidden tests 23/24, typecheck pass) ```diff …```
For each: {"id","solves":0-3,"regressionRisk":0-3,"reviewRounds":n,
           "mergeable":true|false,"note":"one sentence"}

Cost, in plan terms. API list price is the wrong unit for a fleet on subscriptions; what we spend is a percentage of a weekly window. Codex models run one at a time, cases in parallel inside each model, and the weekly used-percent in the session logs is read before and after each batch, so each model’s delta is its own. The Claude CLI reports an API-equivalent dollar figure per run plus the plan’s utilisation, so the lane’s weekly delta over its summed dollars prices each attempt. Both plans cost about 130 CAD a month, so one percent of a weekly window is worth about 0.30 CAD.

// one weekly-window percent, in money
const cadPerPct = monthlyPriceCad / (4.345 * 100);          // ≈ 0.30 at 130/mo
// Codex: measured per model batch
const codexPctPerAttempt = (usedAfter - usedBefore) / attempts;
// Claude: calibrate the lane once, then price each attempt
const pctPerUsd = (weekAfter - weekBefore) * 100 / laneApiUsd;
const claudePctPerAttempt = attemptApiUsd * pctPerUsd;

Break-even with a reviewer. A cheap builder is only cheap if review and escaped bugs stay cheap too. With p the share of attempts the judge calls mergeable, and a reviewer that catches a share q of bad PRs while falsely blocking a share f of good ones:

// plan $ per GOOD merged PR, and bad merges that slip past per 10 good ones
const perGood  = (buildCost + reviewCost) / (p * (1 - f));
const badPer10 = 10 * (1 - p) * (1 - q) / (p * (1 - f));
// always-on: window % one busy seat burns per hour, and seats one week keeps busy
const pctPerHour = pctPerAttempt / (minutesPerAttempt / 60);
const seats24x7  = 100 / (168 * pctPerHour);

The reviewer figures are assumptions for now: a “cheap” reviewer at 0.05 CAD that catches half and blocks a fifth of good work, a “strong” one at 0.20 CAD that catches 85% and blocks a tenth. A sibling trial replays real PRs through candidate reviewers to replace them with measurements.

Runs log

Every run appends a row. v1 of the harness had four candidates and no guards; v2 added the effort ladder, the control and discrimination checks, the base-relative typecheck, and plan-meter costing.

Results

Pilot, 2026-09-30: one small case, eight candidates

C1 is the smallest case: a task mirrored to the calendar appeared twice in a count, and the fix drops the calendar event only when its linked task is already counted.

Sonnet, luna at medium and 5.6-luna produced the shipped fix almost line for line. Haiku and luna at low effort made the same wrong turn: they dropped every calendar event with a linked task instead of only the ones already counted, so a task moved to tomorrow loses its block. Two vendors converging on one plausible mistake says something about the issue text.

Seven cases, seven candidates, 2026-09-30

Three Claude models and four Codex configurations sat all seven cases, and one judge run scored all 49 attempts together. A fifth Codex configuration, astra at medium effort, was the one the fleet itself ran. It was cut at the start of its batch, because the cheaper batches had already shown what it would cost and the fleet needed the rest of the week.

Mergeable per case:

The shape is a staircase with one landing. Opus was mergeable on every case; on the billing case the judge called its attempt nearly the same as the shipped design. Astra at low effort was the only other model to get the billing case right and missed only the sync race. Then comes a flat landing at five of seven, where Sonnet and two luna configurations sit together and fail the same two hard cases in the same way. On the billing case they all let an event take ownership of a subscription it should not. On the sync race they guard on a timestamp alone or read the token at the wrong moment: the judge noted that 5.6-luna guarded on a last-updated timestamp alone, which misses two edits in the same millisecond, a case the shipped tests deliberately exercise. Nobody but Opus solved the sync race.

Luna at low effort beating luna at medium is not a finding. It is one attempt per cell. In the pilot, luna at low effort failed C1; in this run it solved C1 outright. Luna at medium’s low hidden-test total is almost all C2, where it took a different route from the shipped fix and passed 74 of 134.

Haiku is the surprise, and not a good one. Two of seven mergeable, and it cost more than Sonnet: 0.61 API-equivalent dollars per attempt against 0.39, because it works harder for a worse answer. It averaged 41 turns per attempt against Sonnet’s 17, and read nearly seven times as much cached context. On the hardest case it ran ten and a half minutes, left its debugging scratch files in the tree, and scored a zero. Cheaper per token is not cheaper per fix.

What it cost

Each plan has its own weekly window, so the honest comparison is how far each batch moved its own meter:

The whole Claude lane moved its meter by one point, and the Claude CLI’s dollar figures split that point across the attempts. The Codex meter has no dollar figures, so a batch that does not move it only gets an upper bound. The bounds still say plenty. Both luna batches together stayed under one point for fourteen attempts. Astra at low effort spent five points on seven, at least eight times as much per attempt and probably far more, and that is astra at its cheapest setting. Both meters read whole percents, so a one-point move on the display means anything from almost nothing to just under two points. The Claude figures rest on exactly one such move, so treat them as illustrative estimates, not measurements: the true cost could be far smaller, or up to about twice as large. The ranking between astra and luna does not lean on that. A five-point move means at least four real points, a move of zero means under one, and that gap survives any rounding.

For an always-on fleet the useful number is the share of the week one busy seat burns per hour. Fresh attempts understate a long-lived seat, so treat these as best cases. Opus burned about 1.3% per busy hour and Sonnet about 1.1% (illustrative, from that same one-point move), so one Claude week keeps about half a seat busy around the clock. The two medium-effort luna batches spent 72 attempt-minutes under one point, below 0.8% per busy hour, so a Codex week keeps at least two thirds of a seat busy. Astra at low effort burned 9.5% per busy hour, about a tenth of a seat. That is the incident in miniature: the same plan, run on astra, is gone in an evening.

Break-even with a reviewer

Plan CAD per good merged PR, and bad merges per ten good, using the reviewer assumptions above. The luna rows count build cost as zero, because the meter could not see it; the bound adds at most 0.08.

Read the right-hand columns. Any builder at five of seven ships four bad PRs per ten good with nobody reviewing. Behind a strong reviewer it is down to under one, and it costs more per good PR than Opus does alone, because the reviewer is now the main cost and the weaker builder needs it more. Within one plan the cheapest builder is the one that needs the least review, and on the Claude plan that is the expensive one.

Across plans the arithmetic changes, because the two weekly windows are separate budgets. Every hour astra spends is an hour the Codex plan cannot spend on anything else, and luna gives that plan more than ten times the hours.

What we changed

The fleet’s two Codex builders moved from astra at medium effort to 5.6-luna at medium effort. 5.6-luna ties Sonnet on mergeable cases and hidden tests, has the lowest risk score of the luna configurations, added no new type errors, and stays below what the Codex meter can see. Every PR a Codex seat builds is already reviewed by a Claude Opus seat and by the automated review before it can merge, which is the strong-reviewer column above. The Codex reviewer stays on astra, because review is the judgement-heavy step and its diffs are short. The two hard case types, billing ownership and sync races, are what Opus is for.

What went wrong building the harness

The results are the second half of the story. The first half is six ways the harness produced confident nonsense.

1. The wrong Node. A Node 22 on the PATH shadowed the project’s pinned Node 24, under which every root test fails, so every candidate scored zero and the report looked like a verdict. Hence the control check: if the shipped tests do not pass at the shipped commit, nothing else may print.

2. Binary files from the sandbox. Codex’s sandbox wrote a compile cache into the working tree, and its null bytes rode into the diff and crashed the judge, because the prompt went through argv. Untracked files are now filtered for null bytes and size, and prompts go in on stdin.

3. Pre-existing type errors. The base already failed the typecheck, so “typecheck: fail” was true of every attempt and meant nothing. The column became “no new type errors”, counted against the base’s own list.

4. Name-coupled hidden tests. In the Claude lane, every model passed the identical number of hidden tests on C3 and C7, 5 of 10 and 71 of 97, because the shipped tests import a helper by the name and location the shipped implementation chose. Opus put the same helper where the issue suggested and lost five tests for it. Two Codex configurations later happened to pick the shipped names on C3 and scored 9 and 10 of 10 there, which proves the point from the other side. Hidden tests partly measure “picked the same names”; that is why the judge sees the diff.

5. Plan-meter granularity. Both vendors’ meters report whole percents, and one attempt costs a fraction of one. Batching one model at a time is the only way to get a delta that belongs to one model, and it is still coarse.

6. The trial nearly measured the wrong tier. The personal Codex config on the machine was on the highest effort and the priority tier, the configuration that caused the incident. The harness now pins the fleet’s own profile: default tier, medium or low effort, compaction at 80K tokens.

Limitations, honestly

Seven cases, one attempt per cell. There is no variance estimate and coding agents are not deterministic; the next step is at least three repeats per cell, reported as a range. Until then “5 of 7” means five of seven, not 71%.

The judge is Opus, which is also a candidate, and Opus came top: exactly what self-preference would produce. The pilot is weak evidence against it (the judge gave Codex luna and Sonnet a 3 and its own model a 2 on the same case), and so is the hidden-test column, where Opus also leads, but neither is a check. The check is a second judge from the other family on the same shuffled diffs, plus agreement between verdicts and hidden tests on the discriminating cases.

The cases are fixes the fleet itself shipped, the kind of issue it is good at, which flatters every candidate. A harder set would include issues the fleet failed and a human finished.

Fresh-context attempts understate a long-lived seat’s burn, since a real seat re-sends hours of context on every call, so the always-on figures are ceilings, not estimates. And the reviewer side of the break-even table is still assumptions.

This is a regression suite for models

The frozen part is the point. The seven cases are seven (base commit, merge commit, test files, issue text) tuples plus the exact prompt each candidate received. None of that changes when a vendor ships a new checkpoint under the same name, so the same command on a schedule is a drift detector: if Sonnet’s mergeable count on these seven moves, something moved, and the dated rows above are where it shows.

To add a model. Add one candidate line (name, vendor, model id, effort), run it alone against the same cases, and judge it jointly with the saved attempts from earlier runs so the judge sees it beside them. Add a runs-log row and a dated results row. A different vendor needs only a different launcher.

To re-run a model. Same cases, same prompt, same graders, new date. Compare it with its own earlier row before comparing it with other models.

To test a skill or prompt change. Hold the model fixed and vary one thing: the task brief, the repository’s agent instruction files, or a skill the agent can load. Record a hash of the varied artefact in the runs-log row and compare the two rows for that model within a short window, so the difference is the change and not drift. The same hash on a drift re-run proves the brief was unchanged.

What would change our mind

We would stop sending hard cases to Opus if a cheaper model reached six of seven mergeable over three repeats; astra at low effort already did it once, at a price the plan cannot carry around the clock. We would move the Codex builders back off luna if its PRs needed more review rounds in the fleet than the two this table predicts, or if a luna PR merged a bug a stronger builder would not have written. We would distrust the judge if a second-family judge disagreed on more than one verdict in seven, or if its mergeable calls disagreed with hidden tests on discriminating cases. We would retire a case once two consecutive runs showed every model passing its hidden tests identically. And we would redo the cost model the day either meter reports tenths of a percent.

Takeaways