Module 06 · signal fire · Incentives ·
Rules for Mixed Company
Mechanism design for populations of humans and machines — where intentions are free and incentives are everything.
Brussels, 2036. The directive is four years old and working exactly as written. Compliance among registered firms: 96 percent. The regulator’s dashboard is green. The regulator’s problem is the other dashboard — the one estimating that a third of the sector’s actual activity now runs through agent collectives that are not firms, not registered, and not, under the current definitions, anyone’s legal problem. The rules did not fail. The population they were written for left.
The mechanism
A rule is not an instruction; it is a move in a game where every other player responds. The 2016 edition’s politics chapter made that argument about human institutions — incentives over intentions, unintended consequences as the default, Goodhart’s law as a conservation principle. The baseline adds one parameter and changes the game’s tempo: machine participants (B1) probe rules exhaustively, exploit them instantly, and route around them at software speed, while the rules themselves still update on institutional time (B6).
That asymmetry produces the response surface this module’s toy lets you feel. Too loose, and visible harms accumulate. Too strict, and activity migrates beyond the rule’s reach — the gray zone — where harms return unmeasured and untaxed. The optimum is interior, ugly, and mobile: it shifts whenever the population’s composition shifts, which under B1 is constantly.
You cannot write a rule for what a population intends. You can only price what it does — and mixed human–machine populations reprice faster than any rulebook.
The appreciating skill is the mechanism designer’s habit of mind: assume every rule will be gamed by the fastest optimizer subject to it; design for the response, not the intention; prefer rules that are cheap to update over rules that are satisfying to write; and measure the gray zone, because that is where your policy’s real grade is posted.
⏵ Toy 06 · Policy as a game
innovation (5-year avg): 63 / 100
harms (5-year avg): 29 / 100
activity beyond the rule's reach (year 5): 42 / 100
year 1: innovation 68 · harms 19 · gray zone 8%
year 2: innovation 67 · harms 22 · gray zone 13%
year 3: innovation 64 · harms 28 · gray zone 20%
year 4: innovation 61 · harms 34 · gray zone 30%
year 5: innovation 57 · harms 43 · gray zone 42%
Setting 5/10 with 40% agent share. Past the sweet spot: each extra notch of strictness now buys less safety on paper and pushes more activity where no rule reaches. Try 2.
Geneva, 2036: the treaty is drafted by one delegation of humans and ratified in a chamber where forty thousand agent-swarms hold observer status. They observe very politely. The amendment they want has been circulating for six milliseconds.
- Take one rule you own — a team policy, an API quota, a review gate — and red-team it: “if an optimizer wanted to satisfy this rule’s letter while gutting its purpose, how?” Fix the rule or add the measurement.
- Instrument your gray zone: for one process you govern, find out what fraction of the real activity happens outside the process. That number, not compliance, is your grade.
- Practice writing updateable rules: version them, date them, and pre-commit a review trigger (“revisit when X exceeds Y”). Rules with no update path are future gray zones.