Setting the switches
The skilled job everything so far has been building toward: deciding where each individual switch goes — and what evidence moves one.
Every chapter so far has been building toward one decision, taken many times. Not should we run a dark factory — that question has no good answer because it is asked at the wrong granularity. The real question is per loop, and there are hundreds of them.
The danger is setting every switch to the same mode. All dark: you tear it down four months later. All lit: review becomes the bottleneck and nothing ships. The hard, skilled job is deciding where each switch goesprimary.
What earns a loop the dark
A loop earns fully automated status only if its check is cheap, runs at high frequency, and cannot be faked outprimary. All three, not two of three.
Cheap means you can afford to run it on every change without thinking about it. High frequency means it runs constantly, so a failure points at something recent and small. Cannot be faked out does the real work: the oracle must not be gameable by the thing it is checking. An agent that writes both the code and the tests can satisfy the tests without satisfying anything else — “assert true” is a passing test.
| Earns the dark | Never will |
|---|---|
| A nightly job that fixes exactly one lint violation and opens one pull request short enough to read. | Unattended changes to auth, billing, or a public API contract. |
| Green-or-red oracles: it compiles, the types check, the property holds, the links resolve. | Anything whose failure mode is a subtle production bug no test sees. |
| Mechanical migrations with a deterministic script as the reference implementation. | Decisions that shape a year of work: architecture, boundaries, the data model. |
| Work with a small blast radius and a cheap revert. | Anything where being wrong is expensive and only a person will catch it. |
| Loops of three to ten steps, where the check runs against a small delta. | Factual integrity: whether a citation is honest, not merely whether it resolves. |
That last row deserves a pause, because it is the case where the check can be faked out by the same distribution being checked. Whether a citation resolves is machine-decidable. Whether it is honest is not. A generator that invents plausible text can invent plausible sourcesworkspace. This site’s own answer is the register you are reading: a machine can enforce that every claim points at a registered entry, which is a real guarantee, and it cannot enforce that the entry is fairly characterised, which is why the colophon claims only the first.
How the reference dark factory earned it
StrongDM’s Software Factory, publicly described in February 2026, is the one well-documented Level-5 example, and it operates under two rules: code must not be written by humans, and code must not be reviewed by humansprimary.
The interesting part is the machinery built to make those rules survivable:
- Scenario-as-holdout. End-to-end user stories stored outside the codebase, like a machine-learning holdout set, so the agents cannot see, overfit or game the tests that judge themprimary. This is a direct structural answer to “cannot be faked out”.
- Digital Twin Universe. Behavioural clones of Okta, Jira, Slack and Google Workspace, built by agents from public API docs and shipped as self-contained binaries, so scenarios can run thousands of times an hour without rate limitsprimary.
- “Satisfaction.” A probabilistic success metric (what fraction of observed trajectories likely satisfy the user) replacing boolean “tests pass”primary.
Read that list as a price rather than a feature set. It says: darkness is available to teams willing to build an entire second system whose only job is to be un-gameable. And the literal price is stated too: a benchmark of at least $1,000 per day per engineer in tokens, roughly $20,000 a monthprimary, which Simon Willison flagged as the make-or-break question for whether the pattern generalises beyond well-capitalised teamsprimary.
Where the human goes
The person never left the factory; they movedprimary.
Agents own the inner loop: investigate, implement, test, report. Engineers own the outer loop: decide whether it is the right change at all, verify the diagnosis, approve, and carry the consequences of being wrongprimary.
The boundary between the loops is evidence: diffs, tests, logs, and a brief explanation connecting themprimary. That word is doing load-bearing work. What crosses the boundary should not be a claim that the work is good; it should be the material from which someone can decide. A handoff that says “all checks passed” is a claim. A handoff that says “here is what ran, here is what it found, and here is what nothing checked” is evidence.
You are not down on the line writing changes. You are at the end of the line, designing it and guarding the gate. Robots are fine in the dark; humans need to see what they are doingprimary.
The playbook
The staged version, which is where the field survey and the frame independently land. What follows is the spine. The working version lives in What to do on Monday: six steps inside the stages, each stage with an explicit condition for moving on and a condition for stopping.
Stage 1: build a light factory now (0–3 months). Standardise on a spec-driven framework: Spec Kit for greenfield rigour, OpenSpec for brownfield, BMAD only where a full multi-agent team is justified and its token cost is acceptableprimaryreputable. Adopt a coding-agent harness with a disciplined context layer, subagents and worktree isolationprimary. Keep the human merge gate. Move on when agent-authored PRs routinely pass CI in one or two rounds and review time per PR drops below your human-authored baseline.
Stage 2: invest disproportionately in the verifier (2–6 months). Guides plus sensors, quality kept left. Layer computational sensors on every change and reserve inferential ones for post-integrationprimary. Track DORA metrics and cost per useful review. Stop if verification cost per change approaches what you saved on generation; at that point the factory is net-negative and the answer is narrower, more testable tasks.
Stage 3: consider dark only for narrow, richly verifiable domains (6+ months, conditional). Only where you can build behavioural twins of dependencies, hold out scenarios the agents cannot see, tolerate the token spend, and accept probabilistic rather than boolean correctnessprimaryprimary. Start where Stripe and Spotify started: flaky-test fixes, migrations, well-specified featuresprimaryreputable.
The dimmer switch
One last idea, and it outlasts the rest of the book.
Darkness is not a switch you flip. It is the asymptote of the outer loopprimary. The mechanism that gets you there is a retrospective that mines run records for what humans caught after the machine gates passed — the marginal catch rate per gate. When retros show a gate catching nothing for N consecutive runs on a class of work, that gate has earned lights-out there, with evidence rather than appetiteworkspace.
The four properties from §8.1 are worth having as an instrument rather than a paragraph, because the failure mode in practice is asking the question once, about the whole organisation, instead of per loop. Set the switches for one loop’s actual check:
Test a single loop, never an organisation. Set each switch to what the loop's check actually has today, not what the roadmap promises it.
3/4 conditions met
The loop has earned an unattended trial. Start narrow, log every gate, and keep an automatic park condition.
The check can still be gamed. Automate execution, but keep approval lit until the oracle is genuinely independent.
The verification system cannot yet absorb full autonomy. Narrow the loop or improve the oracle before increasing speed.
Back-pressure rule: you cannot automate judgment you cannot observe.
The dimmer page carries a standalone copy of this instrument that holds its setting in the URL, so a configured dimmer can be passed around in a workshop.
And the same loop tells you which gate must stay lit forever. In the case studies, the generated sites are disposable; the factory’s own templates and knowledge documents are its brownfield. That is where comprehension debt would actually accumulate, so edits to the factory itself stay human-gated, permanently, by the same logic that lets everything else go darkworkspace.
Darken one gate at a time, on evidence you collected before you needed it, and always on a named class of work rather than across the board. If you cannot say what the gate caught in the last N runs, you are not making a decision — you are making a bet, and the bill for that bet arrives in month four.