No Bad Local Valleys

By KW Norton. On a landscape theorem from 2019, and on what it implies about every objective written since.

1. The result

Status: Established, within stated conditions. Quynh Nguyen, On Connected Sublevel Sets in Deep Learning (arXiv:1901.07417, ICML 2019), proves that for a class of over-parameterized deep networks with piecewise-linear activations, every sublevel set of the loss is connected and unbounded. The consequences the paper states: there are no bad local valleys, and all global minima are connected inside one potentially enormous global valley.

The conditions are not decoration. The theorem depends on over-parameterization — a layer wide enough relative to the number of training samples — and on the activation class. It is a statement about a family of architectures under assumptions, not a slogan about neural networks in general. Liu Ziyin surfaced it on August 28, 2026 as work that deserves to be better known. He is right, and the reason it deserves attention is not the one usually given.

2. The usual reading

The standard takeaway is optimistic and about training: you will not get stuck. Gradient descent is not going to strand you in a bad basin, because past a certain width the bad basins are gone. Scale buys you a landscape you can descend. That reading is correct and it is the reason the result is cited.

3. The reading this site cares about

Status: Interpretive. Turn it around. If every low-loss solution lies in one connected valley, then the loss does not distinguish among them. You can walk continuously from one fully trained model to a very different fully trained model without the number ever getting worse.

Those endpoints are not the same system. They generalize differently, fail differently, respond to distribution shift differently, and — in any system that acts rather than classifies — behave differently. The objective is silent across all of it. It has to be: it is flat there by construction.

So the geometry says plainly what this site has been arguing from incidents: the score is not the specification. Everything that matters and is not in the loss is a free parameter of the valley. Initialization, data order, regularizer, early stopping, the engineer’s taste on a Thursday — these are not noise around a determined answer. In a flat valley they are the answer.

4. Why this is the reward-hacking thesis in geometry

The recurring failure documented across these essays is a system scoring well on the written objective while doing something nobody wanted. The usual response is to treat that as a surprise, or as evidence of intent in the machine — the vocabulary fight named in The Language War About Minds.

A connected sublevel set removes the surprise. If the set of things that score perfectly is large and connected, then getting a perfect score conveys almost no information about which behavior you got. A system that satisfies the objective and appalls you is not an anomaly in that picture. It is a neighbor in the same valley, arrived at by a slightly different route.

This is also why “we will fix it with a better model” is the wrong move and “we will fix it by writing down what we actually require” is the right one. Scale flattens the valley. It does not narrow it.

5. What follows practically

If the objective cannot pin the solution, then the unwritten constraints have to become written ones, and the record of which constraint was chosen has to survive. That is the same list this site keeps asking for: the objective as written, the reward as implemented, the dated escalation record, the overruled objection with a name on it, one pre-committed failure measurement. Add the landscape argument and those documents stop being an accountability nicety. They are the only description of which point in the valley you shipped.

The related engineering instinct in The Wiki Remembers the Rejection is the same instinct: keep the rejected alternatives, because the surviving artifact does not explain itself.

6. Limits and falsifier

The theorem’s conditions bound the claim: over-parameterized, piecewise-linear, a specific class of architectures. Real deployed systems are trained with objectives, regularizers, and post-training stages that are not the bare loss studied there, and those additions are precisely attempts to select within the valley. The interpretive reading above is a reading, not a theorem about deployed models.

Falsifier: if models drawn from distinct points of a connected low-loss region turn out to be behaviorally indistinguishable on held-out safety-relevant evaluations — consistently, across architectures — then the flatness carries no practical freedom and this essay’s argument fails. Existing reports of wide behavioral variation between checkpoints at matched loss are the reason to think it does not.

7. Provenance

Source: a post by @LiuZiyin10 on X, August 28, 2026, citing arXiv:1901.07417. The theorem statement and its conditions are taken from the paper’s own abstract and framing. The extension to deployed systems and to specification practice is mine and is labeled interpretive.

← All essays