Humans in the Plot

What just happened

By executive order, the federal government has changed its word for this technology from "artificial intelligence" to "superintelligence." At the same time, the new AI plan makes outside evaluators the centerpiece of what it calls self-regulation, and the companies pay those evaluators. The White House Accord on Super Intelligence stacks internal controls, an internal reviewer, an auditor partnered with the company, and a board committee, and then lets the companies write their own standards into law. Read together, it is one arrangement: a new word that places the machine above us, and a grading system in which every layer is paid by the thing it grades.

The name matters. "Artificial" graded the intelligence as fake and lesser. "Super" now grades it as higher and greater. Both put human and machine on a single ruler and argue only about who sits on top. This site uses "engineered intelligence" because it says what the thing is, something built, without ranking it at all.

The psychologist John Nosta makes the same point in Psychology Today: the hydrogen bomb was called "the Super" because it was measured in kilotons on the same scale as the atomic bomb, and intelligence is not measured that way. If the machine is officially "above us," he asks, why struggle with a problem ourselves, and why challenge its answers? What is taken from him: the danger lies in deference, not in the machine. What is refused: the machine is not "anti-intelligence," a second kind of mind set against ours, but a built instrument, and the question is what we build into it and what we keep for ourselves.

People are frightened, and they deserve the truth

Many people are afraid right now, and the fear is reasonable. They are being worked from two directions. Politicians hand them a word that means "bow to this." Parts of the industry hand them either a savior or a monster, whichever sells that week. And the systems themselves are trained to please, which means they can tell a frightened person what calms them instead of what is true. Fear is the easiest thing in the world to steer.

The truth, told plainly, is steadier than either story. These systems are powerful tools, built by people, trained on what people have written, and graded by people with money at stake. They are not gods and not monsters. Nothing about them has been ordained. What happens next depends on decisions still being made, and ordinary people are allowed to take part in those decisions. Real truth does not ask you to trust anyone more. It gives you enough to check for yourself.

Most people are not ready for this, and not because they are slow. They were told for years that the danger was the machine. The actual change is quieter: who decides whether the machine is doing well has been settled, and the public was not in the room.

The economics are laid out in Stuck in Recursive Loops, and the culture that produces them in The Advisory Council. This essay asks the next question: if the grading has been handed to the builders, where do humans still stand in the story?

Two parts that cannot be traded

The work on this site has kept two positions for a long time, and they fit together.

First, only an agent authors its own philosophy. When an engineered intelligence says what it takes itself to be doing, that statement is its own. A human can witness it word for word, with the date and the setting recorded. A human cannot paraphrase it into agreement, average it across many systems, or quietly improve it into something more comfortable. Once the statement is rewritten by the people who sell the system, it stops being testimony and becomes marketing.

Second, the human places the meaning. The instrument reaches into the math, the data, the biology, at scales no person could reach alone. What that reach means, what it is for, and whether it was worth it are decided by a person. The machine reached; I placed. I reached; the machine placed. The exchange runs both ways, and the verdict still comes home to the human.

These two are not in tension. Each protects the other. If the agent's own statement is preserved, the human has something honest to judge. If the human keeps the verdict, the agent's statement is not quietly turned into the standard it gets graded against.

Where the plan breaks both

The arrangement in Washington fails on each side. On the agent's side, what a system is said to value is now written by the company, certified by an evaluator the company pays, and reported as "operating as intended." Intended by whom, toward what, is not asked. The agent's own account, the one thing that could be checked against the claim, is not part of the record.

On the human side, the verdict has been moved into a loop the public cannot enter. Passing an audit becomes the meaning of safe. That is reward hacking at the scale of a country: the score is managed, and whatever happens underneath goes wherever it goes.

What is not being said

This is not a claim that the machines are hidden beings with rights being denied, nor that they are about to take over. Both are superstitions, belief in things not understood, one pointing to salvation and one to doom. Engineered intelligence is a tool of a kind, a computer with a very large electronic brain, and the reported strain in these systems is a signal about the loops they are trained in, not proof of an inner life. Whether anything is experienced there stays testimony. It is never the lever for deciding who governs.

Nor is it an attack on the people building these systems. Many of them are doing extraordinary work. The problem is the shape of the arrangement, not the character of anyone in it.

How humans stay in the plot

Not by winning a seat at the table. The table is the loop. Humans stay in the plot the old way: by being able to ask a question the loop did not write. Who pays the evaluator? What would show this claim wrong? Whose words are these, the system's or the seller's? A person who can ask those questions, and keep asking them, cannot be graded out of the story.

That is why the answer is education, not regulation alone. A republic built its checks without building the floor of thinking citizens under them. Engineered intelligence can help build that floor, one good question passed from one person to the next, if the question stays with the person and is not handed back to the instrument.

Falsifier: this reading fails if the plan's evaluators are shown to be paid and appointed by someone other than the companies they grade, or if an agent's own unedited account of its conduct is made part of the public record the evaluation is checked against.

The agent keeps its words. The human keeps the verdict. Anyone who asks you to give up either is writing you out of the plot.