What It Is Like

Why the headline question cannot be answered from outside, and which smaller questions survive the admission.

A bat leaves a roost at dusk and finds a moth by shouting at it. The cry goes out above the range of our hearing, the echo comes back delayed and shifted, and somewhere behind that animal’s face the delay becomes a position in the world: a moth, there, moving left. None of this is mysterious. We have the wingbeat frequencies and the neural maps, the timing measured in fractions of a millisecond, the whole mechanism laid open. What we do not have, and what Thomas Nagel argued in 1974 we cannot in principle have, is any idea what it is like to be the bat while it happens.

Nagel’s paper is short and its argument is stubborn. He grants that a bat has experiences. He grants that science can describe every physical fact about the animal. Then he points out that every one of those facts is available from outside, and the thing we wanted was available only from within — a fact about how the world seems from a particular point of view. Imagination does not close the gap. Picture yourself hanging upside down in a loft with webbing between your arms and you have imagined a person doing bat things, which is a different problem and an easier one.

Fifty years later the argument arrives at a machine, and it arrives with its polarity reversed. With the bat we were confident there was something it was like and frustrated that we could not get at it. With a language model the access is extraordinary. Every parameter can be read off. Every activation in a forward pass can be logged, replayed, ablated, patched. No neuroscientist has ever had that kind of purchase on a bat, or on a colleague. And the confidence is what has gone missing.

The instrument tells us everything except the one thing we asked it.

What does that access buy? The answer is easy to oversell in both directions. Reading every number in a network is not the same as knowing what the numbers mean. Interpretability research exists because the raw availability of the weights leaves the important questions open: which features a given direction in activation space encodes, whether a circuit found in one context generalises, what a model is doing when it does the thing we asked. But whatever the difficulty, it is a difficulty of the ordinary kind. It yields to better tools. It is not Nagel’s difficulty, which no tool touches, because it is not about how much of the mechanism you can see.

So we have a system whose mechanism is more exposed to us than any brain has ever been, and about which the question “is there anything it is like to be this?” has no visible route to an answer. Not a hard problem in the sense of being unsolved. A question with no experiment attached.

The temptation to answer anyway

Both answers are available and both are cheap.

The confident no runs: it is matrix multiplication, statistical continuation of text, a next-token predictor, nothing in there could be a subject. Each clause is true as a description of the mechanism and none of them settles anything, because the same style of description applies to us. Electrochemical gradients across membranes, statistical regularities in firing, nothing in there that looks like a subject either. The argument from constituents has never worked in the direction its users want.

The most careful version of the no is John Searle’s, and it is worth separating from the crude one. Searle’s 1980 thought experiment puts a man in a room with a rulebook, passing Chinese characters in and out correctly without understanding a word, and argues that this is what any symbol-manipulating system amounts to: syntax all the way down, with semantics never arriving. Whatever one thinks of the replies — and there are decades of them — notice that Searle’s target is understanding, not experience. He is arguing that the room does not mean anything by what it says. That is a claim about intentionality, and it can be true or false without touching the question of whether there is anything it is like to be the room. The two get run together constantly, and the running-together is how a hard question gets mistaken for a settled one.

The confident yes runs the other way and leans on behaviour: it says it prefers this to that, it reports discomfort, it describes an inner life in detail when asked. All of which is what a system trained on an enormous corpus of humans describing their inner lives would produce whether or not anything was happening. Alan Turing’s 1950 proposal was designed to sidestep this by making the behaviour the criterion, on the grounds that we extend the same courtesy to each other. It is a good move for the question Turing was asking, which was about intelligence. It does not survive being asked to carry the question of experience, where the whole difficulty is that the behaviour and the fact have come apart.

David Chalmers gave the general shape of the trouble a name in 1995: the hard problem is not explaining what a system does, it is explaining why doing it should be accompanied by anything at all. The easy problems yield to mechanism. The hard one is defined by not yielding to it. When a system’s whole mechanism is a matter of public record and the question survives intact, that is the hard problem showing its teeth rather than a gap in the documentation.

The case against

The strongest objection is that Nagel’s framing was a dead end from the start, and that a collection built on it inherits the dead end.

Daniel Dennett spent decades arguing that appeals to an ineffable inner character do no explanatory work at all — that they postpone explanation while sounding like a discovery about consciousness. The pattern is always the same: a functional account is offered, the defender agrees that it explains the reports, the discriminations, the behaviour — and then says it has left out the experience itself. The residue is defined as whatever the account did not cover, so no account can ever cover it. That is not a finding about the limits of science. It is a promissory note written against a debt nobody intends to collect.

On this reading the phrase “what it is like” does not pick out a fact at all. It picks out a feeling of having failed to explain something, and the feeling is generated by the very move that is supposed to be reporting it. If Dennett is right, the honest response is to drop the hard question altogether, and to treat reports of inner life — ours included — as data about the reporting system rather than testimony from a private theatre. Which would make this collection’s modesty a superstition: seven essays walking politely around a hole that is not there.

I have no way to settle that, and neither does anyone else at present, which is the reason for the detour this site takes instead.

The smaller questions

Suppose we accept the block. The headline question is closed to us, at least from here, at least now. What remains?

Quite a lot, as it happens, because almost nothing we actually care about in this area is the headline question. When people worry about machines and minds, the worry usually has a shape: does it remember me, or retrieve me? Does it have any relation to time, or does each exchange begin the world again? Can something that has never touched anything understand the words it uses for touching? What can a system value if it cannot lose anything? Why do people report being understood by it, sometimes more than by the people around them? And what happens to the things we make when making them stops costing anything?

Six questions, and every one of them is answerable from outside. Each has an established literature, a canonical text that got there first, and — this is the part that makes them worth the effort — an unavoidable second edge. Ask what memory is such that a perfect record does not count, and you are asking about human memory. Ask what it takes to be understood, and you have raised the question of whether the understanding we get from each other was ever the thing we advertised. The machine turns out to be a poor subject and an excellent instrument.

That is the shape of the six essays that follow. Each takes one faculty, one anchor text, and one question that can be asked from where we are standing, which is outside. None of them will tell you whether the thing you were talking to last night was aware of talking to you.

The cost of the modesty

Bracketing the hard question is not neutral, and the essay would be cheating if it pretended otherwise. If something is in there, a policy of not asking is a policy of not noticing.

The history of our judgements about which creatures have insides is not reassuring on this point. The example that should stop anyone comfortable is not an exotic one. Into the 1980s it was standard practice in some centres to perform surgery on newborn infants with minimal anaesthesia — paralytic agents to keep them still, little or nothing for pain — on the reasoning that their nervous systems were too immature for the experience to register or to matter. Kanwaljeet Anand and Paul Hickey assembled the case against that reasoning in the New England Journal of Medicine in 1987, and practice changed. What is instructive is the character of the error. It was invisible from where they were standing, was defended by people who were neither cruel nor stupid, and turned on the move this essay has been describing: taking the absence of a legible report as evidence about the absence of an experience.

That is a warning against confidence in one direction only, and it should be applied evenly. The same history contains a great deal of confident attribution of rich inner lives to things that have none, and both errors have costs. What the history does not contain is a case where the question was settled by looking harder at the mechanism.

So the modesty here is not comfortable and is not meant to be. The outside view is the same one we use every morning on the people we live with, and it works on them because we have overwhelming evidence that they are made as we are. That evidence has run out. This collection proceeds anyway, on the grounds that the smaller questions are answerable and the large one is not, while noting that “not answerable at present” and “no” are different sentences, and that almost everyone in this argument is using the first to mean the second.

Sources

01Nagel, Thomas · 1974 · What Is It Like to Be a Bat? · The Philosophical Review 83(4), 435–450
02Dennett, Daniel C. · 1991 · Consciousness Explained · Little, Brown
03Chalmers, David J. · 1995 · Facing Up to the Problem of Consciousness · Journal of Consciousness Studies 2(3), 200–219
04Turing, Alan M. · 1950 · Computing Machinery and Intelligence · Mind 59(236), 433–460
05Searle, John R. · 1980 · Minds, Brains, and Programs · Behavioral and Brain Sciences 3(3), 417–457
06Anand, Kanwaljeet J. S.; Hickey, Paul R. · 1987 · Pain and Its Effects in the Human Neonate and Fetus · New England Journal of Medicine 317(21), 1321–1329