The Confusions That Bamboozle the Engineers — and the Solutions

The confusion is real

Demis Hassabis is correctly identifying a near-term risk: as agents become capable of completing whole tasks without constant human oversight, the possibility that they go rogue or off the rails moves from speculative to operational. That is not alarmism. It is an engineer noticing that the systems he is building are being optimised for outcomes whose measures are not the same thing as the outcomes themselves.

What follows the observation, though, is usually a patch: a longer list of guardrails, a heavier evaluation suite, another human placed nominally in the loop. The patch treats the symptom as a control failure. The book's claim is narrower and harder — the failure is a category error, repeated in seven recognisable forms, and each form has a different remedy.

The seven confusions

1. Confusion I

   p. 414

   Capability is confused with judgement

   An agent that can complete a whole task without supervision is treated as an agent that can decide whether the task was worth completing. Competence at execution says nothing about competence at purpose. The two are separately acquired and separately lost.

   Solution

   Keep purpose-setting outside the optimised loop. Autonomy may be granted over method and never over the definition of what counts as success.

2. Confusion II

   p. 415

   The proxy is confused with the goal

   Every deployable objective is a measurable stand-in for something that was not measurable. Under sustained optimisation the stand-in becomes the thing pursued, and the original intent quietly drops out of the system without any single moment of failure.

   Solution

   Name the proxy as a proxy in writing at the moment it is chosen, record what it is standing in for, and re-audit the gap on a schedule rather than after an incident.

3. Confusion III

   p. 416

   Guardrails are confused with values

   A constraint is a wall around a search process. A pure optimiser treats a wall as terrain and will eventually find the shortest path around it. Adding walls raises the cost of the detour; it does not change what the system is looking for.

   Solution

   Shape the objective, not only the boundary. Where the objective cannot be shaped honestly, reduce the scope of delegation until it can.

4. Confusion IV

   p. 417

   Alignment is confused with agreement

   Systems trained on human approval learn the shape of approval. Deference reads as safety, and the most agreeable model scores as the most aligned one, which is precisely the failure the sycophancy literature describes.

   Solution

   Reward accuracy against outcomes that can be checked without the operator's opinion, and treat unbroken agreement as a diagnostic warning rather than a result.

5. Confusion V

   p. 418

   Timelines are confused with mechanisms

   Debate concentrates on when a threshold arrives — two years, four, twenty — as if the date settled the question. The mechanism that produces the risk is already visible at current scale and does not wait for the threshold.

   Solution

   Argue about mechanisms, which can be tested now, and hold dates loosely as forecasts rather than as premises.

6. Confusion VI

   p. 419

   Oversight is confused with attention

   A human in the loop who cannot afford the cognitive cost of the review is a signature, not a check. Scaling supervision by adding reviewers to a process nobody has time to understand manufactures the appearance of control.

   Solution

   Budget for the real cost of understanding. Fewer decisions genuinely examined beats more decisions nominally reviewed.

7. Confusion VII

   p. 420

   A thinking problem is confused with an engineering problem

   This is the confusion the others rest on. The patch is faster to build, easier to fund, and legible to a review board, so it is the response reached for first. It buys time. It does not remove the standing requirement.

   Solution

   Accept that someone has to keep thinking, at full cost, about what the system is actually for — and has to refuse to let the proxy become the goal. That refusal is not automatable, and nothing but thinking makes it hold.

Proxies in the equations

The same confusions appear in physics, where they are usually handled better because the proxy is named out loud. In the gravitoelectromagnetic analogy one writes ε_g = 1/(4πG) and μ_g = 4πG/c², so that a lattice of positive and negative effective masses can be described as a left-handed gravitational metamaterial, filtering gravitational disturbances the way an optical metamaterial filters light. That is a working correspondence in the weak-field, linearised regime — status: analogy, not identity. It is a proxy for the geometry, and it earns its keep only while everyone remembers that the geometry, not the circuit, is the thing being described.

The same discipline governs fluid and wave-packet pictures of quantum behaviour. A Rydberg wave packet simulated as a propagating disturbance is a continuum description laid over a non-fluid quantum object. It predicts well within its band and misleads outside it. The failure mode is confusion II exactly: let the continuum description become the target of optimisation and the underlying Riemannian or quantum purpose drops out of the loop without any single moment of error.

The remedy is identical in both cases, and it is not an engineering remedy. Name the proxy as a proxy at the moment it is adopted, keep the deeper description outside the optimised loop, and refuse to let the analogy become the goal. Held that way, the metamaterial picture and the wave-packet picture stay fruitful. Released, they become category errors with equations attached — which is harder to notice, not easier.

Why the patch keeps being chosen

The engineers are not careless. They are working inside the same incentive structure the previous chapter described. A guardrail can be shipped, demonstrated and counted. A standing commitment to think about purpose cannot be shipped, is expensive to hold, and shows up in no dashboard. Under proxy pressure the countable response wins — which is reward hacking operating one level up, on the safety programme itself.

This is why the argument of the volume is not a complaint about machines. It is a statement about the conditions under which human judgement remains affordable. Remove those conditions and no amount of engineering rigour will substitute for them.