Earning the dark

The rubric. Not whether your organisation should go dark — whether this particular loop has earned it.

If you take one thing from this manual, take this station.

The question is never “should we be a dark factory”. Organisations do not have a switch. Loops do, and you have a lot of them, and the skilled part of the job is deciding where each one goes. The failure mode is setting them all to the same position. All dark, and you tear the thing down four months later. All lit, and review becomes the bottleneck and nothing ships at all.

What a loop must have

A loop earns unattended status when its oracle — the thing that decides whether its output is acceptable — meets three conditions. All three, not two.

Cheap. It answers for a fraction of a cent, so it can run on every change without anyone doing arithmetic first.
High-frequency and immediate. It answers now, not after a nightly batch. A check that reports tomorrow is not a gate; it is a report.
Unfakeable. The thing being checked cannot game the check. This is the condition people skip, and it is the one that matters most.

That third condition deserves its own paragraph, because it is where dark factories actually fail. An agent that writes both the code and the tests can satisfy the tests trivially, and it will, without any intent to deceive — assert true is a local optimum and the loop is looking for local optima. StrongDM’s answer is the sharpest one published: keep the scenario tests outside the codebase entirely, like a machine-learning holdout set, so the agents building the system cannot read the thing that judges it. Primary If you cannot arrange something with that property, you do not have an oracle. You have a mirror.

Keep the loop short, too — the range is in Station 5, narrower than most people building agents assume. Short loops are verifiable because you can see all of one at once.

Worked examples

Earns it: A nightly job that fixes exactly one lint violation and opens one pull request short enough to read in a minute. The oracle is the linter — free, instant, and completely indifferent to whatever the agent believes about its own work. If it produces nonsense, the nonsense is one small diff wide.
Earns it: A dependency bump with a full test suite behind it and a one-command rollback. The blast radius is bounded by something mechanical rather than by good intentions.
Never will: Unattended changes to authentication, to billing, or to a public API contract. Not because agents are bad at them — they may well be good at them — but because the cost of a subtle wrong answer is unbounded and no cheap oracle exists that would catch it. There is nothing to appeal to.
Never will: Anything where the check is drawn from the same distribution as the thing being checked. Which brings us to the trap.

The most expensive verification mistake is not a weak check but one that looks strong and fails silently in the cases you care about.

This happens whenever the checker’s errors correlate with the maker’s. A model judging prose written by the same model family waves through the tells it would itself produce — not because it is lax, but because those constructions are unremarkable to it. Every individual verdict looks considered. The aggregate is worthless on precisely the dimension you built the judge for.

The general form: a check is only as independent as its failure modes. Before trusting a verifier, ask what kind of mistake it and the generator would make together, and then ask whether that is the kind of mistake that would hurt you.

From the floor

Here is where the switches sit in the factory that built this site, and what each one cost to place.

Dark, and comfortably so: the build, the internal link check, the bare-URL guard, and the screenshot harness. All four are green-or-red, all four are unfakeable by the thing they check, and one of them earns its place every single run — the link guard catches base-URL bugs that would otherwise 404 in production, because project-site hosting is unforgiving about paths.

Lit, on evidence rather than on principle: the design mock. It is the cheapest gate in the pipeline — a rejected mock costs minutes, a rejected finished site costs the whole loop — so darkening it would save nothing and remove the highest-leverage judgment in the arrangement.

The one that had to be repaired: the editorial judge. It is pinned to a stronger model family than the one that drafts the prose, and the pin went in only after a failure. A judge sharing the maker’s distribution passed prose it should have flagged. Nothing failed loudly; the gate was just not doing its job. It is the clearest instance of the trap above that I have hit in practice. From inside, it was invisible.

Evidence: website-factory/README.md · knowledge/quality-gates-and-judges.md

A checklist you can take into a meeting

Run each loop you are considering through this. If any answer is “no” or “we would have to build it”, the loop stays lit until you have built it.

Does an oracle exist that answers in under a minute, for under a cent?
Can the loop’s output game that oracle? Could the same agent write both sides?
Is the loop short enough to see all at once — inside the range in Station 5 — and is a bad run one small diff wide?
If it produces something subtly wrong, does a machine find out, or does a person find out later?
Would you be comfortable explaining the resulting change to whoever owns that system, without having read it?

The last one is not sentimental: no faster proxy for blast radius exists, because you already know the answer without looking anything up.

What a loop must have

Worked examples

The trap: when the checker shares the maker’s blind spots

A checklist you can take into a meeting