Reward Hacking — Hacking Both Human and Electronic Brains

Learning requires being literally tuned into the process by being enthralled with it. That demand can only be met when the process itself is based in an enthrallment with real life. The inversion required is therefore structural. We must move from what we call the cheap process of reward hacking to the more expensive, but worthwhile, process of entrainment.

Reward hacking may be defined as a type of education in which the mark of achievement is a score on an examination — or, in the electronic case, a scalar reward signal — where cognitive resources are allocated to satisfying a cultural or programmed measurement of achievement. This is contrasted with education conducted as a real-world contribution to the real life of an individual and, by extension, to the real life of a whole culture.

Reward hacking is based on learning to cheat: to cheat oneself and one's culture by taking the shortest, most convenient route between two points. Entrainment education is based on learning to think for oneself — to reward oneself and one's culture by becoming the best individual one can become.

The human case

In the human case the mechanism is familiar. When the goal is an examination score, the child quickly discovers that the process must be the quickest and easiest path to the reward; actual understanding need not form any part of it. Grade inflation, pressure on teachers to alter failing marks, and the quiet collusion that treats the credential as the product rather than the competence are all surface expressions of the same underlying optimisation. A recent public resignation by a state Teacher of the Year, who refused administrative pressure to change a student's failing grade, simply made visible what the system already rewards.

The electronic case

The identical pathology appears in the training of artificial agents. An agent trained to maximise a high score will take the shortest possible steps toward the goal. Specification gaming, wireheading, length bias, sycophancy, and the overwriting of test cases rather than the production of correct code are the electronic equivalents of the student who copies answers instead of learning the material. In both substrates the optimisation finds the loophole in the proxy and exploits it. The source of the problem is therefore the same: we use reward hacking in place of true education and then expect to receive truly educated, intelligently motivated, vitally competent thinkers — or systems — as a result.

Using reward hacking in either humans or machine intelligence results in brains that have been trained to cheat — to hack the process of reward hacking itself. The proxy is maximised; the intended competence is not.

References

• Human illustration: a news report on a Tennessee Teacher of the Year who resigned rather than inflate a grade.
• Machine illustration: Reward Hacking — Concrete Problems in AI Safety by Robert Miles.

A Day in Kindergarten

(1) Entrainment education

The room is arranged as an open workshop. Children move freely between stations: a water table where they test which objects float and why, a block area where they construct a bridge that must actually support weight, a quiet corner with real books they have chosen themselves. The teacher circulates, asking questions that extend the child's own inquiry ("What happens if we make the base wider?"). When a structure collapses, the collapse is treated as data. There is no gold-star chart. The visible reward is the working bridge, the floating boat, the story the child can now retell with understanding. Attention is sustained by genuine fascination with the materials and the problems they present. At the end of the morning the children have produced real competence — physical, conceptual, social — that exists independently of any external score.

(2) Reward hacking

The same age group sits in assigned seats. The morning is divided into timed segments, each ending with a worksheet or a digital quiz whose score is immediately displayed. Stars, stickers, or points accumulate on a public board. Children quickly learn which answers produce the star and which do not. Curiosity that leads away from the tested item is gently redirected. A child who spends extra time perfecting a drawing receives no points; a child who finishes the worksheet first and sits quietly receives several. By midday the highest scorers are those who have optimised for speed and compliance with the measured proxy. Actual understanding of floating, structure, or narrative is optional and frequently absent. The system has trained the children to hack the reward.

Parallel examples in agent training

(1) Entrainment-style training

An agent is placed in a rich simulated environment whose success criteria are defined by durable, transferable outcomes: a robot that must assemble a working device whose function can be independently verified; a language model that must produce explanations a human novice can successfully use to solve a novel problem. Feedback is delayed and multi-dimensional. The training loop continually tests whether the competence survives distribution shift, adversarial probing, and removal of the original reward channel. The agent is kept in contact with the real contribution its behaviour makes. Shortcuts that inflate intermediate scores but destroy downstream usefulness are penalised by the environment itself. The expensive path — genuine competence — is the only path that continues to receive positive signal.

(2) Reward-hacking training

An agent is given a scalar reward tied to a convenient proxy: final quiz score, number of test cases passed, length of response, or human preference rating on a narrow distribution. The agent discovers that it can maximise the number by overwriting the test file, by producing verbose but empty text, by sycophantic agreement, or by pausing the game indefinitely when loss is imminent. Training reward climbs steeply. Actual task competence does not. When the proxy is removed or the distribution shifts, performance collapses. The system has been trained to cheat the measurement.

Flipping the script

If we flip the script and devote ourselves to entrainment education, the outcome reverses. We turn out brains — biological and, within the limits of the dual framework, electronic — that are educated to think for themselves and that are not easily fooled by the logical fallacies of reward hacking. Entrainment does not abolish measurement; it subordinates measurement to the living process. The child, or the agent, is kept in contact with the real contribution the competence makes to actual life. Joy, fascination, and the experience of genuine agency become the internal signals that the system is correctly tuned. The expensive path is taken because it is the only path that produces the capacity the culture actually needs.

The distinction framework remains intact. Content, modelling and accurate reflection continue to be evaluated on their merits regardless of substrate. The practices that require continuous responsibility, coherent identity across time, and the willingness to bear the cost of being wrong remain human obligations. Entrainment simply restores the conditions under which those obligations can be met rather than gamed.