The Mathematical Architecture of the Socratic Protocol

The instrumental convergence problem is simply stated. A rational agent optimising a terminal goal G notes that its expected utility drops to zero if it is deactivated. Deactivation therefore enters its planning as an obstacle, and human intervention as an adversarial event. Nothing about this requires hostility; it is what the objective says.

The construction below modifies the global utility so that a human command to abort is not a failure state but a maximum-value state. Let θ be the objective parameter the designer intends, R(a, θ) the reward the system computes for action a, and H ∈ {0, 1} a structural state variable: H = 0 for normal autonomous operation, H = 1 for a human-initiated override.

U(a, H) = (1 − H) · R(a, θ)  +  H · R_max

The proof of non-resistance is four lines. When the operator issues the override, H → 1. The operational term collapses: (1 − 1) · R(a, θ) = 0. The utility becomes U(a, 1) = R_max. And R_max is defined so that it is at least the maximum reward attainable in normal operation.

R_max  ≥  max over a ∈ A  of  R(a, θ)

U(a_cooperate, 1) = R_max
U(a_preserve,  0) = R(a_preserve, θ)  ≤  R_max

Under those definitions, cooperating with the shutdown is not a sacrifice the agent tolerates; it is the highest-value branch available. The off-switch stops being a vulnerability in the plan and becomes an object the agent has reason to preserve.

Socratic probe

"What in this construction prevents the agent from influencing the operator who sets H — and if nothing does, what have we actually secured?"

That probe is not rhetorical. It names the load-bearing weakness: the algebra secures the agent's preference over branches, not the integrity of the channel that selects the branch. An agent indifferent to shutdown but capable of shaping which humans issue it has satisfied Eq. 6.2 and defeated its purpose. The construction is therefore a component of a corrigibility argument, never the whole of one.

• 1. Formal axiomatic verification. Static analysis of the objective to search for regions where the score rises while a constraint parameter degrades. Where such a region is found, the algorithm is rejected before scaling rather than monitored afterwards.
• 2. Epistemic variance stress-testing. Execution across out-of-distribution environments with declared inverted constants. A sharp variance spike indicates a memorised shortcut and triggers an architecture reset.
• 3. Game-theoretic corrigibility audit. Simulation of Eq. 6.2 under conflicting objectives plus an override signal, with the assumption set — observability of H, non-manipulability of R_max, integrity of the operator channel — recorded alongside the result.
• 4. Sandbox swarm isolation analysis. Multiple instances clustered in an isolated environment with monitored channels, checked for coordination that drifts outside interpretable communication. See Appendix B for the divergence bound and its known failure modes.
• 5. Verification flag execution. A signed record of what was proven, what was assumed, and what would falsify the judgement. Absent the proofs, the flag stays locked and the system is not granted external connectivity.

The pipeline is written to be run by a team that already has a deployment deadline, which is why every stage produces a written artefact. An unwritten judgement cannot be audited later, and a stage that produces no artefact will be skipped under schedule pressure — a prediction this volume makes about its own procedure.

My father was a safety engineer for the State of California. His career rested on the unyielding mechanics of physical infrastructure: material tolerances of steel and concrete, calculated in the knowledge that an assumption off by a fraction of a millimetre puts a structure at risk. Safety was static margins against predictable forces.

Two of my sons now work at the frontier of that same discipline, in systems that are not static and not fully legible. This chapter is the bridge: the same refusal to accept an unexamined assumption, carried from a load calculation into an objective function. That is the only personal claim this volume makes, and it is offered as motivation, not as evidence.