Chapter 8 · the tool

Analysis of Competing Hypotheses

ACH is an eight-step procedure for holding several explanations against each other at once instead of testing them one at a time — the chapter Heuer calls “arguably the most important single chapter” of his own (Ch. 1, p. 2).

The habit it breaks

The ordinary way of working is to settle intuitively on the likeliest answer, then read the traffic for whether it supports that answer — the satisficing of Chapter 4 (p. 95). It finds plenty of support. The trouble is that most of that evidence sits just as comfortably under explanations you never wrote down and so never refuted (p. 96). Willpower will not fix it: tracking three to seven hypotheses against every item of information exceeds what most people manage unaided (p. 96).

The eight steps

1. Identify the hypotheses worth considering, brainstorming with a group of differing perspectives before anyone judges likelihood.
2. List the significant evidence and arguments for and against each.
3. Build a matrix — hypotheses across the top, evidence down the side — and work out which items are diagnostic, meaning they help tell the hypotheses apart.
4. Refine it: reconsider the hypotheses, delete evidence with no diagnostic value.
5. Draw tentative conclusions by trying to disprove hypotheses rather than prove them.
6. Test how sensitive the conclusion is to a few critical items.
7. Report the relative likelihood of all the hypotheses, not only the leading one.
8. Identify milestones that would show events taking a different course.

Paraphrased from Heuer’s outline, Ch. 8 · p. 97 — the wording is this site’s, the procedure his.

Step 3 is the one people get wrong. Heuer flags it as the most important element and the furthest from intuition (p. 100), with a memory aid: there you work across the rows, one item of evidence at a time, asking how consistent it is with each hypothesis; only in step 5 do you work down the columns. Reading down a column first is how you confirm a favorite. Reading across a row exposes evidence that fits every hypothesis equally — evidence that feels like proof and carries no diagnostic weight.

Reject rather than confirm

The principle reverses the ordinary direction of proof: you are after the hypothesis that survived your attempts to kill it. Keep unproven hypotheses alive until they are actually disproved, and do not drop the deception hypothesis because well-executed deception leaves nothing to find (p. 98).

“The most probable hypothesis is usually the one with the least evidence against it, not the one with the most evidence for it.”

His worked example is the Indian nuclear tests of 1998. The Intelligence Community had reported that “there was no indication the Indians would test in the near term” — a formulation that cannot distinguish an unproven hypothesis from a disproved one (Ch. 8, p. 108). Under ACH, one hypothesis would have been that India intended to test and was concealing preparations (p. 109). Hence a test for your own drafts:

Before you write “there is no evidence that…”, ask whether, if the hypothesis were true, you could realistically expect to see the evidence (Ch. 8, p. 109).