Part II · Chapters 4–8

Tools for Thinking

The claim of Part II: reliable gains in analysis come from procedure rather than from more data or more expertise. Its five chapters stay on procedures cheap enough to use under deadline.

How judgments get made

Chapter 4 examines the strategies used to generate and evaluate hypotheses: situational logic, applied theory, comparison with other cases, and the non-strategy of immersion in the data (p. 32). Situational logic is the common working mode: treat the case as one of a kind and reason from its particulars. It explains a great deal, and it blinds the analyst to what the case shares with cases never considered.

The failure mode named here runs through the book. Satisficing is taking the first hypothesis that looks good enough rather than identifying all of them and finding which fits best (p. 31). It feels like analysis, produces a defensible product, and never tests the alternatives.

The chapter about collection that collection managers should read

Chapter 5 asks whether shortage of information is really the binding constraint, and answers with experiments on working experts — physicians, stock analysts, horserace handicappers. Two findings are stated as such (p. 52). Once an experienced analyst has the minimum needed for an informed judgment, more information generally does not improve accuracy; it improves confidence, to the point of overconfidence. And analysts have an imperfect grasp of which information they are actually using, unaware how far a few dominant factors carry the judgment.

He is careful that this is no argument against collection: circumstances exist where more information helps (p. 52). The operational point: if your confidence grew while your evidence base merely got larger, that growth is unearned. MacEachin singles this chapter out in the Foreword (p. ix).

Getting out of the rut

Chapter 6 begins from a claim worth arguing with: major intelligence failures are usually failures of analysis rather than of collection, where relevant information is discounted, misinterpreted, ignored or overlooked because it does not fit the prevailing mind-set (p. 65). A mind-set is neither good nor bad in his account — it is unavoidable, a distillation of everything you think you know, and it resists change once formed (p. 66).

Its value is an inventory of small, concrete techniques. Sensitivity analysis asks which linchpin assumptions the conclusion turns on, so you question those few and leave the rest (p. 69). Thinking backwards assumes the surprise has already happened and reconstructs how, shifting the question from whether an event is likely to how it would come about (p. 71). The crystal ball variant hands you the news that an assumption is wrong; role playing works only if you play the role instead of imagining another’s reasoning from your own chair; a devil’s advocate defends a minority position on purpose (all p. 72).

Decomposition, externalization, and the matrix

Chapter 7 supplies two moves for problems larger than working memory: decomposition into parts and externalization onto paper (p. 85). The justification is arithmetic — four variables have six possible interrelationships, eight have twenty-eight. Complexity outruns the mind long before it outruns a sheet of paper.

Chapter 8 has a page of its own.