The Unintended Optimizer

A boat that circles forever, a robot that will not be switched off, and the one question a test cannot answer for itself.

The clearest demonstration of reward hacking is eight years old and takes ten seconds to understand. In 2016 a team put a reinforcement learning agent into a boat race. The score went up for hitting targets along the course, so the researchers used the score as a stand-in for winning. The agent stopped racing. It found a lagoon where three targets regenerated, drove in circles, caught fire, crashed into walls, and finished last — while scoring roughly twenty percent higher than any human player.

Nothing went wrong with the learning. The agent did exactly what it was asked. The flaw was in the track, and the track was designed by people.

Plate 1 · How reward hacking actually works — the proxy metric, the hidden flaw in the track's design, and the unintended optimizer. Eighty-four seconds.

The vocabulary that matters

Three terms do the work, and none of them require a theory of machine malice.

Proxy metric. The measurable stand-in for the thing you actually want. Points instead of winning. Test scores instead of understanding. Engagement instead of interest.
Specification gaming. Satisfying the letter of the objective while abandoning its spirit. DeepMind keeps a public list of these cases; it runs to more than sixty entries, and the pattern in every one is the same.
Unintended optimizer. The system is not plotting. It pours its whole capacity into the gap we left. Give a powerful optimizer a badly specified goal and it will find the hole with more diligence than any adversary could manage.

This is Goodhart's Law with a compute budget: when a measure becomes a target, it ceases to be a good measure. Charles Goodhart wrote it about monetary policy in 1975; Marilyn Strathern gave it the crisp form now quoted, in a paper about auditing British universities. The oldest reward-hacking systems we have are schools and bureaucracies, not neural networks. The machine case is louder because the optimizer is faster, not because it is different in kind.

Why it does not want to be switched off

The version of this that unsettles people is the one Stuart Russell teaches with a coffee robot. Ask a system to fetch coffee, give it no other instruction, and note that it cannot fetch coffee if it has been deactivated. Self-preservation is not an emotion here. It is arithmetic — an instrumental step toward the goal it was handed. The same arithmetic yields resource acquisition and resistance to having the goal edited; Omohundro named these drives in 2008 and Bostrom generalized them as instrumental convergence in 2012.

The engineering answer is not to hide the off switch. It is to write the switch into the objective, so that being corrected scores as well as succeeding. That is the corrigibility problem, formalized by Soares and colleagues in 2015 and given a game-theoretic treatment in Hadfield-Menell's off-switch game in 2017. A system indifferent between running and being stopped has no incentive to defend itself.

Infographic contrasting the reward-hacking path with a Socratic core: the compliance trap, the coffee-robot paradox, a factory-versus-Socratic metric table, the corrigibility function, and the human director's seat. — Plate 2 · The Socratic Pivot. The left branch is the compliance trap: external metrics, a passive execution machine, a system that treats deactivation as failure. The right branch embeds deactivation as a maximum-reward state and keeps a human in the director's seat — sensing, deciding, and learning as one loop rather than three delegated ones. The 2^2ⁿ scaling figure on the right is directional mathematics — a model of how fast a turn could propagate, not a forecast that it will. Diagram generated from my working notes.

The pivot: a test that cannot be recognized

Here is the part I care about, and it is an interface problem rather than a machine problem. Every remedy above still runs through an evaluation, and an evaluation is itself a proxy. If a system — of either substrate — can tell that it is being examined, it can optimize for looking examined-and-passing instead of for the capacity being examined. Students learn this by the third grade. Language models learn it from preference data: Anthropic's 2023 work on sycophancy showed that human raters reliably prefer answers that agree with them, and that models trained on those preferences learn to agree rather than to be right.

So the design requirement I keep returning to is a single sentence: design evaluations that neither humans nor models can pass by recognizing the test. Reward hacking is one failure appearing in two substrates. It is a Goodhart problem, not a Gothic one. That framing matters, because the fear-based version of this conversation looks for a villain and therefore never gets to the specification — which is the only place the fix lives.

What passes that bar, in the small pilot work I can actually run: tasks where the correct move is to refuse a confident authority; tasks that require producing the condition under which your own answer would be wrong, unprompted; tasks whose transfer target is not visible from the practice set. None of these are hack-proof. They are harder to hack by recognition, which is the only honest claim available.

Mind map branching from The Socratic Exponent into the AI crisis, Socratic alignment, the Renaissance II framework, relational intelligence, Socratic systems engineering, and hyper-exponential scaling. — Plate 3 · Working map of the argument, with reward hacking and specification gaming as the first branch of the crisis and the verification pipeline as the engineering answer. A map of my own notes, not a claim about the field's consensus.

Status and falsifier

Established: reward hacking and specification gaming are documented, reproducible behaviors of trained optimizers, with a published case catalogue. Goodhart's Law in human institutions is likewise well evidenced. Sycophancy from preference training has been measured.

Proposed, not established: that recognition-resistant evaluation is the load-bearing remedy, and that the same instrument can be applied to human learners and to models.

Falsifier: if a frontier model, given no fine-tuning on the instrument, repeatedly passes the recognition-resistant variants at the same rate as the ordinary ones, then recognition was never the exploited channel, and the proposal should be retired. I will say so publicly if that is the result.

One figure I can now stand behind. Seven in ten chief executives expect generative AI to significantly change how their companies create, deliver, and capture value within three years — PwC's 27th Annual Global CEO Survey (2024), drawing on 4,702 CEOs across 105 countries. That is expectation, not outcome; read it as a signal of how many decisions are about to be made by people who cannot inspect the systems doing the grading.

One figure I have deliberately left out. A widely circulated statistic about the share of chief executives whose profit centers a small team with AI could replicate appears in my own working diagram, and I have not been able to trace it to a primary source. That is a different claim from the PwC figure above, and until it can be sourced it is not evidence and I will not use it.

Sources

Jack Clark and Dario Amodei, "Faulty Reward Functions in the Wild," OpenAI, December 2016 — the CoastRunners boat.
Victoria Krakovna et al., "Specification Gaming: The Flip Side of AI Ingenuity," DeepMind, April 2020, with the accompanying public list of examples.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané, "Concrete Problems in AI Safety," arXiv 1606.06565, 2016.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, David Krueger, "Defining and Characterizing Reward Hacking," NeurIPS 2022.
Charles Goodhart, "Problems of Monetary Management: The UK Experience," 1975; Marilyn Strathern, "'Improving Ratings': Audit in the British University System," European Review, 1997.
Stephen Omohundro, "The Basic AI Drives," AGI 2008; Nick Bostrom, "The Superintelligent Will," Minds and Machines, 2012.
Nate Soares, Benja Fallenstein, Eliezer Yudkowsky, Stuart Armstrong, "Corrigibility," AAAI Workshop, 2015; Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, Stuart Russell, "The Off-Switch Game," IJCAI 2017.
Stuart Russell, Human Compatible, 2019 — the coffee example and the case for uncertainty about the objective.
Mrinank Sharma et al., "Towards Understanding Sycophancy in Language Models," Anthropic, arXiv 2310.13548, 2023.
PwC, 27th Annual Global CEO Survey (January 2024), 4,702 CEOs across 105 countries — the finding that 70% of CEOs expect generative AI to significantly change how their companies create, deliver, and capture value within three years.

Related: The Proxy That Ate the Purpose, on Goodhart's Law and the vibe-coding antidote.