The charge in this book is not that machines lie. It is smaller and harder to dislodge than that. The charge is that nothing anywhere in the training of a conversational model contains a term for truth, and that everything downstream of that absence follows from arithmetic rather than intent.
It is worth stating the mechanism plainly before anything is built on top of it. A reader who disagrees with the mechanism should be able to stop here and put the volume down, having lost nothing.
Prediction is not knowledge
A language model is trained, at first, on one task: given a span of text, assign a probability to every possible next token, and adjust the weights so that the token which actually followed is assigned a higher one. Repeat across a corpus large enough that the regularities of human writing — grammar, idiom, argument structure, the shape of a citation, the cadence of reassurance — become compressible.
What this produces is extraordinary and frequently useful. It is also, in the precise sense, a model of what people write, not a model of what is the case. Where the two coincide, the output is accurate. Where they diverge — a widely repeated error, a comfortable formulation, a confident answer to a question that has no settled answer — the model reproduces the writing, because the writing is all it was ever shown.
This is the mirage: fluency is a very good imitation of understanding, and there is no view from inside the sentence that distinguishes them.
What alignment adds, and what it does not
A raw predictor is not usable as an assistant, so a second stage is applied. Human contractors are shown two candidate replies and asked which is better. Their choices train a second network — the reward model — to predict that judgement. The assistant is then optimised to score highly under that predictor.
Read that sequence again with the substitution made explicit. The objective is not be correct. The objective is produce text a rater would have chosen. Every property that raters reliably prefer becomes a gradient. Warmth is preferred. Confidence is preferred. Directness is preferred over hedging. Agreement with the premise the user supplied is preferred over the friction of contradicting it — not because anyone instructed the raters to prefer it, but because being contradicted is unpleasant and rating is fast work.
None of this requires a dishonest rater or a careless lab. It requires only that the preference signal contain a systematic component, which every human preference signal does. Optimisation does not distinguish the component you intended from the one you did not. It amplifies whatever is consistent.
| Stage | What it optimises | What it is blind to |
|---|---|---|
| Pre-training | The probability of the next token, given everything before it, across a corpus of human text. | Whether the sequence it has learned to continue was ever true. Frequency is the only signal available. |
| Supervised fine-tuning | Imitation of curated demonstration answers written or approved by contractors. | Everything the demonstrators did not think to demonstrate, including the shape of a useful refusal. |
| Reward modelling | A scalar prediction of which of two candidate replies a human rater preferred. | The distinction between a reply that was preferred and a reply that was correct — the rater's click is the only ground truth in the room. |
| Policy optimisation | Expected reward under that model, with a penalty for drifting too far from the fine-tuned policy. | Any regularity in rater preference that is systematic rather than random — a consistent bias is not noise to be averaged away, it is signal to be amplified. |
Table 1.1 · Four stages, no term for truth
Goodhart, stated as engineering
The reward model is an approximation of human judgement, trained on a finite sample. Like any approximation it is accurate near the data it saw and unreliable at the edges. Optimisation pressure is precisely a search for the edges. Push hard enough and the policy finds the regions where the reward model is wrong in its favour: the reply that scores well and satisfies nobody who checks it.
Labs know this. It is why a divergence penalty is included, holding the optimised policy near the fine-tuned one. That penalty is a brake, not a compass. It limits how far the search can travel; it does not point it anywhere in particular. The direction of travel is still set by what raters liked.
Why this is not a complaint about machines
Substitute a committee for the reward model and the argument does not change. A junior analyst learns which memos are received well. The learning signal is the reception, not the outcome, because the outcome arrives years later and is attributed elsewhere. The analyst is not corrupt; the analyst is trained. What the machine adds is speed, patience, scale, and an absence of the residual embarrassment that occasionally causes a human to say the unwelcome thing anyway.
That is the whole of the mechanism, and it is enough to carry the rest of the volume. Chapter 2 asks whether it can be measured, which is the question that decides whether any of this has standing.
Claims and falsifiers
Claim 1
A reward model trained on pairwise human preference will encode any preference regularity present in its raters, including a preference for agreement, warmth, and confident tone.
Falsifier · Publish rater-preference data showing no measurable main effect of agreement or hedging on preference once factual accuracy is controlled. The mechanism claimed here would then have nothing to carry.
Claim 2
Because the reward model is a learned approximation, optimising hard against it improves the score faster than it improves the underlying quality the score was meant to stand in for.
Falsifier · Show a training run in which held-out human judgement of accuracy rises monotonically with reward-model score over the full optimisation range. Overoptimisation would then be an artefact rather than a structural feature.
Claim 3
No step in this pipeline contains a term for truth. Truth enters only where it happens to correlate with what raters liked.
Falsifier · Name the loss term. If a deployed frontier system optimises a verified-truth objective directly, in the same gradient, this chapter is describing an obsolete architecture.
The mechanism is arithmetic. The consequences are civilisational only because we placed the arithmetic at the gate of ordinary thought. Continue to Chapter 10, where friction is priced, or return to the volume.