The Stag and the State · Part 2 of 3
The Shadow of the Future Is Shrinking
In 2011, a teacher named Sarah Wysocki was fired by the District of Columbia school system. Her principal rated her highly. Parents liked her. What ended her career was a number: a low score from IMPACT, the district’s evaluation system, whose value-added model estimated her contribution to student test results. The estimate was noisy, the formula was proprietary, and there was no meaningful way to contest it. Cathy O’Neil opens Weapons of Math Destruction with her case because it is so ordinary. Nothing dramatic happened. A system produced a score, the score produced a decision, and the decision was final.
This essay is the second part of an argument that began with The High-Speed Stag Hunt, which examined why autonomous software agents default to low-value, risk-free transactions. The mechanism there was game-theoretic: when identities are disposable and memory is short, cooperation stops paying. This time the subject shifts from agents negotiating with agents to people living inside scoring systems. The claim is that the same mechanism operates on them — from the opposite side of the interface.
A term worth defining precisely
Robert Axelrod, whose 1984 book The Evolution of Cooperation remains the standard reference, borrowed a phrase for the thing that makes cooperation rational: the shadow of the future. It means, simply, the weight that tomorrow’s interactions cast over today’s choices. If I expect to deal with you again — and expect you to remember how I behaved — then cheating you today is expensive. If today is our last round, cheating is free.
Axelrod’s tournaments produced a result that has survived four decades of scrutiny: cooperation does not require virtue. It requires conditions. Interactions must be durable, counterparts must be recognizable, and the future must matter enough relative to the present. Strategies like tit-for-tat win not because they are nice but because the game is long. Shorten the game, and the same strategies — the same people — turn defector.
It is worth pausing on how unusual this framing is. Most public argument about algorithmic systems is moral: the systems are biased, or extractive, or unfair, and the people who build them should feel worse than they do. Axelrod suggests a colder and more useful question: what game are these systems making people play?
The stag hunt, restated for humans
The first essay in this series used the stag hunt — a coordination game in which two hunters can jointly take a stag (high reward, requires trust) or individually settle for a hare (low reward, no trust required). Software agents, it argued, default to the hare: a counterparty that can vanish and reappear under a fresh identity offers no future to cast a shadow, so the rational agent takes the safe trivial payoff and abandons the valuable cooperative one.
Now stand the situation on its head. The agent chose the hare because its counterparty was disposable. The scored human stops cooperating because they themselves have been made disposable — and they know it.
Consider what a modern scoring system does to Axelrod’s three conditions, one at a time.
Durability. A gig-work driver whose rating drops below a threshold is deactivated — not disciplined, not warned, removed from the game. O’Neil’s catalogue is full of these endings: teachers cut by value-added models, job applicants filtered by personality scores, defendants sentenced with recidivism estimates. In each case the decision is terminal from the perspective of the person scored. There is no next round to play, and everyone involved knows it.
Recognizability. Tit-for-tat requires that the other side can see you, remember you, and change its behavior in response. A scoring model does not see a counterpart; it sees a feature vector. You cannot build a reputation with it in any meaningful sense, because it was not designed to reciprocate. The relationship runs one way, like a mirror that only the other side can look through.
The weight of the future. This is where Shoshana Zuboff’s The Age of Surveillance Capitalism adds something the game theorists did not anticipate. Zuboff describes markets in behavioral futures: predictions about what you will click, buy, and want, sold in advance to whoever bids. One need not accept her whole theoretical apparatus to notice the structural point. Cooperation depends on an open future — the possibility that tomorrow could go differently depending on how we treat each other today. A prediction market in your behavior is, precisely, a machine for closing that future: it prices tomorrow now, and the more accurate it gets, the less your future conduct is treated as yours to choose. What the stag hunt needs, the futures market pre-sells.
Put the three together and the conclusion is uncomfortable but hard to escape: algorithmic societies do not make people worse. They make cooperation irrational by shortening the shadow of the future — for the scored, not just the scoring. The driver who games the rating system, the teacher who teaches to the test, the applicant who keyword-stuffs a résumé are not moral failures. They are people responding correctly to a one-shot game someone else built around them.
Reputation systems genuinely do extend cooperation among strangers, and the mechanism is Axelrod’s. But look at which systems produce the cooperative effect. An eBay rating is symmetric: both sides score each other, both can respond publicly, both carry their history into a market with alternatives. A credit report, for all its flaws, is legally appealable: statutes like the American Fair Credit Reporting Act oblige the scorer to disclose, investigate, and correct. These systems resemble repeated games because they preserve the grammar of one — memory plus voice plus exit.
The systems O’Neil documents share the memory and delete the rest. The teacher score was proprietary: no disclosure. It was asymmetric: the district scored her; she could not score the district, nor audit the model, nor carry her rating to a competing evaluator. And it was final: no investigation, no correction, no next round. The distinction that matters runs not between scored and unscored societies but between appealable, symmetric reputation — which lengthens the shadow of the future — and unappealable, asymmetric scoring, which shortens it while borrowing reputation’s vocabulary. The word “score” covers both, which is convenient for the people selling the second kind.
An unpaid debt from part one
Honesty requires revisiting the first essay’s proposed solution, because this series should hold itself to the standard it applies to others. Part one argued that agent markets need collateralized identity and programmatic escrow: to join the high-value hunt, an agent posts a bond it forfeits on defection. Readers acquainted with cryptocurrency will recognize this as staking and slashing — a mechanism that ecosystem has run at scale for years, with results mixed enough to be instructive.
Three failure modes are well documented. Capital lockup: bonded participation prices out anyone who cannot afford idle collateral, so the trust mechanism doubles as a wealth filter. The oracle problem: someone must decide that a defection actually occurred before the bond is burned, and that someone becomes the system’s real authority — reintroducing the trusted judge the collateral was meant to replace. Griefing: adversaries who can trigger slashing conditions cheaply can destroy honest participants’ stakes at a profit, or simply for spite.
Notice what these failure modes have in common: each one recreates, inside the “trustless” mechanism, exactly the asymmetry this essay has been describing. The oracle is an unappealable scorer. The capital requirement is a filter on who gets to play at all. For software agents backed by firms with treasuries, these may be acceptable costs, and part one’s architecture likely still stands for its intended domain. But as a template for human systems it fails, and it fails on this essay’s own axis. A society where people post bonds to be allowed to cooperate is not a lengthened shadow of the future. It is a pawnshop.
The repair for human systems is procedural, not financial: disclosure rather than deposits, appeal rather than escrow, symmetry rather than slashing. This suggests the two halves of the series describe two different games — agents may need collateral because they cannot yet receive due process; humans need due process precisely because most of them cannot afford collateral.
What would change my mind
The mechanism makes two testable predictions.
First: where scoring systems add genuine appeal and repair channels — disclosure of the model’s reasons, a working dispute process, restoration after error — measured cooperation with the system should rise, and gaming should fall, without any change in enforcement. The credit domain, where dispute rights are statutory, is the natural first dataset; a rigorous comparison across gig platforms with and without deactivation appeals would test the claim directly. If appealable systems turn out to be gamed more than unappealable ones, this essay is wrong.
Second: systems that pre-commit to finality — no appeal, no restoration — should exhibit the stag-hunt signature: participants investing minimally, hoarding options, optimizing the metric rather than the task. Historically, this pattern has tended to surprise the system’s designers, who expected accountability and got compliance theater instead.
The shadow of the future, in the end, is less a metaphor about virtue than an engineering parameter — and we are currently tuning it toward zero for the people with the least power to object. The third part of this series will look at what happens when the institutions doing the tuning — the platforms that own the venues — become the closest thing the economy has to central planners.