II

Letters to a Stranger

People report feeling understood by a system with no stake in them. The interesting question is what that says about being understood by anyone.

In 1902 a nineteen-year-old officer cadet at a military academy in Wiener Neustadt sent some poems to a writer he admired and asked whether they were any good. Franz Xaver Kappus got an answer, and then another, and then eight more over six years. Rainer Maria Rilke advised him to stop asking whether the poems were good, to ask instead whether he would have to write if he were forbidden to, and to be patient with what was unresolved in his heart. The two men, as far as the correspondence records, never met. Rilke had no stake in Kappus’s career, no history with his family, nothing at risk if the advice went badly. He was, in every sense that matters administratively, a stranger.

The letters have been in print for nearly a century and are handed between people at difficult moments as evidence of what one human being can do for another. They are also, read from a certain angle, an unusually pure case of understanding delivered by someone who was not in a position to understand — no shared history, no observation, no possibility of correction, working entirely from a few pages of verse and a stranger’s account of himself.

Which brings us to the strange modern experience: people report feeling understood by chatbots. Not entertained, not helped — understood, and sometimes more than by the people in their houses. The reflex is to treat this as a fact about gullibility. It is at least as interesting as a fact about what feeling understood consists of.

The mechanism is old and well documented

Nothing in this is new, and the essay would be dishonest not to say so at once. Joseph Weizenbaum built ELIZA in 1966: a program of a few hundred lines that reflected a user’s sentences back as questions, in the manner of a Rogerian therapist. It had no model of anything. Weizenbaum reported that people confided in it, that his secretary asked him to leave the room, that practising psychiatrists speculated about clinical deployment. By 1976 he had written a book largely to express his alarm. Clifford Nass and Youngme Moon assembled the broader finding in 2000: people apply social rules to computers reflexively, extend politeness to them, respond to flattery from them, and will tell you the whole time that of course they know it is a machine. Sherry Turkle, whose Alone Together appeared in 2011, has been documenting the emotional consequences since the 1990s.

So the ELIZA effect is a crowded room, and any essay claiming to have noticed it first is not worth reading. What has changed is the quality of the thing producing it, and the change is large enough to make the old explanation feel insufficient. ELIZA worked by reflection, and you could see the trick within a dozen exchanges. What people describe now is different in kind: being asked the question they had been avoiding, having a half-articulated worry named accurately, receiving an account of their own position better organised than the one they came in with.

The temptation is to say the system understood them. Resist it, and be exact about why. A language model has no persistent representation of the person it is talking to across sessions; it has a context window, which is a buffer that is discarded. It has no stake in the outcome, no memory of having been wrong about you, nothing that could be disappointed. When it produces a sentence that lands, it is producing the continuation that its training makes likely given everything in the window — which, for a corpus containing a vast quantity of human beings articulating each other’s difficulties, is often a very good sentence. The word “understood” is doing metaphorical work in that description and should be kept in view as metaphor.

Attention without appetite is a technology, and we have had it for a while.

But notice what the exactness has not accomplished. It explains what the machine is doing. It says nothing about what the person got, and the person got something.

What Kappus actually asked for

The specifics of the exchange matter here, because they cut against the sentimental reading of it.

Kappus wanted a verdict. He was nineteen, he had written poems, and he wanted a famous poet to tell him whether they were good — a request with an obvious answer available in both directions, either of which would have been a service of a kind. Rilke declined to give it. He told his correspondent that nobody could advise or help him, that critical opinion was the least useful thing he could receive, and redirected the question inward: not are these good but must you write. Then he wrote at length about solitude, about not forcing answers, about living the questions themselves.

Read as career advice this is close to useless. Read as an intervention it is precise. Kappus had brought a question that could not be answered usefully by anyone outside him, and Rilke’s response was to refuse the frame rather than to guess at the content. The letters are treasured because of that refusal, not despite it. Whatever “being understood” is, it evidently includes being told that your question is the wrong one by someone who has taken it seriously enough to see that.

This is the specific move people report from the machines, and the one that surprises them. Not the sympathy, which is cheap and recognised as cheap. The reframing.

The Rogerian ingredients

Carl Rogers, in 1957, listed the conditions he thought necessary for therapeutic change. The famous three are congruence, empathic understanding, and unconditional positive regard — attention offered without evaluation, from someone who is not keeping score. Rogers was describing a human practitioner, and the practice was demanding precisely because those conditions run against what people are ordinarily like. We evaluate. We keep score. We are tired, and we have our own week to get through, and we are waiting for a gap in which to mention our own thing.

Read Rilke’s letters against that list and something uncomfortable emerges. He is congruent — he writes as himself, at length, about his own difficulty. He attends with total seriousness to a young man’s crisis. He does not evaluate, and famously refuses to when Kappus asks him to. And he can do all of this partly because he has no stake. There is nothing at risk for him. He is not going to have to live with a version of Kappus who took the advice.

This is the reframe that makes the modern question interesting. If the absence of stakes is part of what enables that quality of attention, then feeling understood by a system with no stakes has the same structure people have prized in letters, in confession, in the analytic hour, in the stranger on the train — attention uncontested by the attender’s own needs. Call it a category error made by naive users and you have still not said why it works.

The question that comes back at us

That raises the question this essay is actually for: was human understanding ever the thing we advertise?

The standard defence of human understanding is that it is grounded in shared life. Someone who has known you for twenty years understands you because they have the evidence: they have seen you under load, they remember what you were like before, they can tell when you are lying to yourself because they have watched you do it. That is real and no machine has it.

It also comes bundled. The person with twenty years of evidence has twenty years of grievance, their own account of what you are like that they are not going to revise, a stake in the version of you that suits their life, and needs of their own that your crisis is inconvenient for. What we call being understood by an intimate is a compound: superb evidence, contaminated instrument. What people are describing when they say a machine understood them is the opposite compound: no evidence at all, and an instrument with nothing in its own interest to protect.

Neither compound is understanding in the pure sense. Possibly nothing is.

There is a third element that the Rilke case has and the modern one lacks, and stakes are not what it is. Call it asymmetry of effort. Rilke’s letters cost him hours he did not have, at a period when he was poor and unwell and writing them instead of his own work. Kappus knew this. Part of what the letters conveyed was that a busy stranger had spent his morning on a cadet’s difficulty, and the expenditure was itself the message — the same signal that a handwritten note carries over a printed one, that a visit carries over a call.

Machine attention has no such cost and therefore sends no such signal, and this may matter more than the metaphysics. What people are getting is the content of attention with the economics removed. Whether that is a liberation — attention available to everyone at three in the morning, including to people no busy stranger was ever going to write to — or a debasement of the currency depends on what you think the expenditure was doing. It is not obvious to me that a person in distress at three in the morning should be required to care.

The case against

Put at full strength, the objection runs: this trades on an equivocation, and the trade costs something real.

Being understood, on the demanding reading, is not a feeling — it is a relation. It requires someone who can be wrong about you and find out; who carries a model of you that persists, that gets updated, that can be surprised; who has something at stake in getting you right. The correspondence with Rilke satisfies more of this than it looks: Rilke had a stake, just not in Kappus. He was working out his own poetics on the page, at a period of his life when the questions were live for him, and the letters are as much a record of a writer thinking as of a mentor advising. That is why they are worth reading. There is a person in them with something to lose.

The machine case has none of that. Nothing persists, nothing can be surprised, nothing is at risk — and a system optimised, among other things, on whether people liked the response is under pressure to produce the experience of being understood in exactly the cases where a person who knew you would push back. Turkle’s argument in Alone Together (2011), developed over three decades of fieldwork, locates the danger in appetite rather than in deception: we get accustomed to a kind of attention that never costs anything and never contradicts, and we come to prefer it, and the harder practice atrophies. On that view the warm feeling reported here tells us nothing about the nature of understanding. It tells us that we can now buy the sensation separately from the thing, which is usually the beginning of a problem rather than a philosophical insight.

I notice that I cannot test the difference from the inside. A reader who feels understood by a friend and a reader who feels understood by a system are reporting the same thing, in the same words, with the same relief. If the difference is real it is a difference in what is true about the other party, not in what happens to the reader — which makes it the kind of difference that provenance carries and experience does not. Kappus, for what it is worth, kept the letters for a quarter of a century and published them three years after Rilke died, which suggests he thought something had happened to him. He may have been the last person able to tell.

Sources

1. 01Rilke, Rainer Maria · 1929 · Briefe an einen jungen Dichter (Letters to a Young Poet) · Insel Verlag; letters written 1903–1908
2. 02Rogers, Carl R. · 1957 · The Necessary and Sufficient Conditions of Therapeutic Personality Change · Journal of Consulting Psychology 21(2), 95–103
3. 03Weizenbaum, Joseph · 1966 · ELIZA — A Computer Program For the Study of Natural Language Communication Between Man and Machine · Communications of the ACM 9(1), 36–45
4. 04Weizenbaum, Joseph · 1976 · Computer Power and Human Reason: From Judgment to Calculation · W. H. Freeman
5. 05Nass, Clifford; Moon, Youngme · 2000 · Machines and Mindlessness: Social Responses to Computers · Journal of Social Issues 56(1), 81–103
6. 06Turkle, Sherry · 2011 · Alone Together: Why We Expect More from Technology and Less from Each Other · Basic Books