Every stage below carries two lines that matter more than its contents — a condition for moving
on, and a condition for stopping. A playbook without a stop condition is a sales pitch.
Stage 1 — Build a lit factory
0–3 months
Standardise on a spec-driven workflow so the unit of work is a reviewable artefact rather
than a conversation. Primary Give agents a
disciplined context layer — one file per repository stating how to build, test and check it,
and worker instructions that read that file instead of hard-coding tools. Isolate parallel work
so runs cannot collide.
Keep the human merge gate. All of it. This stage is about making agent output reviewable
at volume, not about reducing review.
- Ready to move on when agent-authored changes routinely pass CI on the first or
second attempt, and review time per change has dropped below your human-authored baseline.
- Stop if you are shipping more and nobody can describe what changed last week. That
is comprehension debt arriving early. The cure is smaller, better-scoped work rather
than more tooling.
Stage 2 — Spend disproportionately on the verifier
2–6 months · the important one
This is where the leverage is, and almost nobody budgets for it, because verification does
not demo well.
Layer computational sensors that run on every change: type checkers, linters, structural
fitness rules, mutation testing where correctness matters. Reserve inferential sensors — an
LLM judge, an architecture review — for after integration, and wire them as advisory rather
than as gates. Primary Keep quality left:
a guide that prevents a class of mistake is worth more than a sensor that reports it.
Then measure the thing nobody measures:
- Cost per useful review, not reviews performed. A verifier that is mostly
right is still expensive if the rest trains people to stop reading it:
20–40%
of machine review comments that are noise
as of 2026 Vendor
- Marginal catch rate per gate — what each human gate still finds after the machines
have signed off. This is the only number that can retire a gate later, and you have
to start collecting it now, while the gates are still there.
- Defect escape rate, because throughput without it is a vanity metric.
- Ready to move on when you have at least one loop whose oracle is cheap, immediate
and impossible for the generator to game — and months of data on what your humans still
catch downstream of it.
- Stop if verification cost per change approaches what you saved on generation. The
factory is then running at a loss; the fix is to narrow the work until it is testable,
not to check less of it.
Stage 3 — Consider dark, narrowly
6+ months · conditional
Note the scope: consider it for a loop, never for an organisation. Take a candidate
through the Station 06
checklist and be strict about the third condition — whether the generator could game the
oracle.
The published route to running this way at scale requires holdout scenarios the agents cannot
read, behavioural clones of your dependencies so scenarios can run in bulk, a tolerance for
probabilistic rather than boolean success, and a token bill of
≥ $1,000/day/engineer
stated token-spend benchmark for running this way
as of 2026-02 Primary
Start where Stripe and Spotify started: flaky-test repair, mechanical migrations, dependency
bumps, well-specified changes behind rich tests. Low stakes, high coverage, small blast
radius.
- Ready when the marginal catch rate for this class of work has been zero for enough
consecutive runs that the zero means something.
- Stop if you cannot say who finds out when this loop is subtly wrong. That is not a
tooling gap; it is the absence of an oracle.