Game Theory and Cooperation

The previous module treated a single mind deciding under its own limits. But few decisions are made in isolation. The moment two bounded agents must each anticipate the other, a new layer of structure appears. Game theory is the formal study of that layer, and in the last decade its oldest puzzles have acquired a new kind of player: the machine.

Strategic Interaction: Game Theory Meets AI

Two burglars are caught near the scene and interrogated in separate cells; neither can hear what the other says. Each has the same choice: stay silent — cooperate with your partner — or betray them: defect. Scored in the points the lab below uses (higher is better), the four possible outcomes look like this:

Read one cell to learn the table: if you defect while your partner cooperates, that is the lower-left cell — 5 points for you, 0 for them. Now check your own best move. If your partner cooperates, defecting pays 5 against cooperating’s 3; if your partner defects, defecting pays 1 against 0. Whatever the other does, betrayal scores higher; a move that is best against every reply is called a dominant strategy. So two rational players end at 1 point each, while 3 each sat on the table. Neither can improve by changing strategy alone: mutual defection is the game’s Nash equilibrium Nash equilibrium A situation in a game where no player can do better by changing strategy alone, given what the others do. It is stable, but not necessarily good for anyone. .

“Rational” may sound like a step backwards, one module after Module 3 retired the perfect calculator. It is a deliberate simplification, not a claim about real minds: assume each player always picks its best reply, and the analysis shows where a game’s traps lie — here, one that catches even flawless reasoners. How bounded players stray from that baseline is the rest of this module.

Each player would do better by trusting the other, and neither can afford to: that is the dilemma in the Prisoner’s Dilemma Prisoner’s Dilemma The classic game in which each player does better by betraying the other, yet both end up worse off than if both had cooperated. It is the standard model of why cooperation is hard. . And the trap generalizes: whenever your best action depends on what someone else does, you are in strategic interdependence, and game theory is its formal study. (Game theorists name the four payoffs Temptation 5, Reward 3, Punishment 1, and Sucker’s payoff 0; any game with T > R > P > S is a Prisoner’s Dilemma.)

Add up each cell and the totals are 6, 5, 5 and 2: the players’ choices change the size of the pie, not only how it is cut. That makes the Prisoner’s Dilemma a non-zero-sum game, which is why cooperating is worth anything at all. In Rock-Paper-Scissors, scored +1 for a win and −1 for a loss, every cell adds up to zero — one player’s gain is exactly the other’s loss. That is a zero-sum Zero-sum A situation where one side’s gain is exactly the other side’s loss, so the total stays fixed. Many real interactions are not zero-sum: both sides can win, or both can lose. game, with nothing to cooperate for.

Rock-Paper-Scissors has a second lesson. Always play rock, and an opponent who notices plays paper and wins every round; any choice or pattern the other side can learn gets exploited. The one safe approach is to be unpredictable — each option at random, a third of the time. A choice randomized with set probabilities is a mixed strategy Mixed strategy In game theory, choosing between moves at random with set probabilities instead of always playing the same move, as a penalty taker varies the side they shoot to. , and in games of pure opposition the equilibrium often lies there. Penalty takers and goalkeepers, varying corner and dive, play one.

The dilemma looks inescapable — but real people have memory, and they meet again. So which strategy is best for a game repeated indefinitely? Robert Axelrod ran that question as a computer tournament around 1980, and the winning strategy, Tit-for-Tat Tit-for-tat A strategy for repeated games: cooperate first, then copy whatever the other player did last time. Simple, forgiving and hard to exploit, it won Robert Axelrod’s computer tournaments around 1980. , was simple: cooperate first, then mirror whatever the opponent did last. Nice, retaliatory, forgiving, and clear.

Then machines learned to play. AlphaZero (2017) mastered chess, Go and shogi through pure self-play — millions of games against copies of itself, with no human examples — and found strategies centuries of human play had missed. Pluribus (2019) beat professional poker players at an imperfect-information game, where no player sees the others’ cards. (The full story is in The AI Revolution.)

But when large language models — AI systems like ChatGPT that generate text by predicting the most likely next words, examined in depth in The AI Revolution — entered game-theoretic settings, the results were more nuanced. Decades of lab experiments had established the human baseline: many people — about half the subjects in a classic public-goods experiment — are conditional cooperators, and many are altruistic punishers — they cooperate when others do, and pay real costs to punish cheaters. Studies found GPT-4 more cooperative than the typical human player in such games. In the repeated Prisoner’s Dilemma, however, it plays what researchers call an “unforgiving” strategy: it cooperates until the first defection, then permanently retaliates. No forgiveness, no recovery. It is, in effect, too rational for its own good: safe from exploitation, and shut out of every later gain from cooperating.

This behavior is prompt-dependent — it shifts with how the question is posed to the model (the prompt). Social Chain-of-Thought prompting — asking the AI to predict the other player’s next move before choosing its own — made GPT-4 coordinate markedly better with human partners, and simply telling it that the other player sometimes makes mistakes made it more forgiving. The model’s architecture (the built-in structure of the system, set long before anyone types a prompt) is unchanged; only the framing differs. This echoes Gigerenzer’s ecological rationality: the “environment” (prompt) determines whether the same system cooperates or defects.

Before you run the tournament below, commit to two guesses: over hundreds of rounds, does Always Defect — which never loses a single exchange — finish first or last? And does GPT-4’s unforgiving strategy beat Tit-for-Tat’s forgiving one?

Does Structure Rescue Cooperation?

If you guessed Always Defect finishes last, the standings just corrected you: in this small pool it lands at or near the top, and the unforgiving strategy outscores the forgiving one — because the lab’s pool contains an unconditional cooperator and a random player to farm. That is Axelrod’s real finding, sharper than the slogan: whether cooperation wins is a property of the population, not of the strategy. Tit-for-Tat won Axelrod’s tournaments because the pool was thick with retaliators — and it won on total points without ever outscoring a single opponent in their own match, since a strategy that only answers defections can at best draw level.

Cooperation needs structure. Reputation systems, repeated interaction, and institutional design create conditions in which cooperating becomes the stable choice rather than the naive one. (Temptation is not even required: in the Stag Hunt — hunt a stag together and feed ten, or hares alone and feed two — cooperation fails from sheer coordination risk, the fear that the other will not show up.)

The Prisoner’s Dilemma’s trap also scales up to many players. Four flatmates each hold 10 coins and may put any of them into a shared pot, which is doubled and split equally among all four. Each coin you put in returns only half a coin to you — two coins, shared four ways — so keeping your coins pays better whatever the others do. If everyone pays in, each ends with 20; if no one does, each keeps 10; if the other three pay in while you hold back, you end with 25. Living off what others contribute is free-riding Free-riding Enjoying a benefit that others pay for while contributing nothing yourself. When everyone reasons this way, the shared benefit is not produced at all. . A shared resource that anyone can draw on and no one is made to pay for — a fishery, a pasture, the atmosphere — sets the same trap: a commons dilemma Commons dilemma The trap around a shared resource that anyone can use and no one is made to pay for, such as a fishery or the atmosphere: each user gains by taking more or contributing less, so together they run it down. . Yet Elinor Ostrom documented communities that have run fisheries, pastures and irrigation systems for generations under rules they made and enforced themselves — structure again, built by the players.

And structure is literal: who plays whom matters as much as the payoffs. In the simulation below, sixty agents sit on a network and play the Dilemma with each neighbor every round. Nobody calculates anything: after a round, an agent may simply copy the strategy of its best-scoring neighbor. Three wirings are on offer — a ring where everyone knows only the people next door, Module 2’s small world (the same ring plus some long-range shortcuts), and links placed at random. Do the shortcuts that make networks efficient help cooperation spread, or help defection invade? Commit to a guess, then try all three structures below.

Run each wiring several times, pressing New seed in between; a single run is noisy and can point the wrong way. Across many runs, the plain ring keeps the most cooperators and the small world tends to keep the fewest. On the ring, cooperators form clusters whose members mostly meet one another, often enough to out-score the defectors on their boundary. A shortcut breaks the seal: it gives a defector a line into the middle of a cluster, where it scores well enough to be copied. The shortcuts that carry a rumor across a network carry defection too. Cooperation, then, is an achievement of structure, not a default of nature.

Notice what the agents never did: calculate. They copied whatever scored well nearby, and strategies spread or died out the way traits do under natural selection. That is evolutionary game theory Evolutionary game theory Game theory without the calculating player: strategies spread when they score well, usually because others copy them, and die out when they do not, much as traits do under natural selection. — game theory with the rational player taken out, where a strategy succeeds by spreading rather than by being chosen. It is the form agent-based models mostly use, and it carries a lesson of its own: the trap found by assuming perfect reasoners also catches players who do not reason at all.

What happens when the structures between us — platforms, feeds, recommendation systems — are redesigned around a different objective entirely? That is the subject of the next module: the contest for attention, and what it does to collective intelligence Collective intelligence The ability of a group to judge or solve problems better than its individual members. It needs the right conditions, such as independent views and a good way to combine them. .