Elon Musk says intelligence is a quantity, and that we are about to have a great deal more of it. The most durable law in physics says intelligence is a cost.

The Cretans had a robot. His name was Talos, he was made of bronze, and he ran a patrol: three circuits of the island every day, on foot, without rest, hurling rocks at any ship that came too close. Depending on which teller you follow he was forged by Hephaestus or handed to Europa by Zeus, and either way he is the oldest machine in the Western imagination: tireless, incorruptible, and not alive.

Except that the people who imagined him could not quite bring themselves to leave it there. Talos, they said, had a single vein. It ran from his neck down to his ankle, it carried ichor, the fluid of the gods, and at the ankle it was closed with a bronze nail. When the Argo came, Medea did not out-fight him. She worked on him until he grazed the ankle against a rock, and the nail came away, and the ichor ran out of him like molten lead, and the bronze giant came down.

That detail is the interesting one. A culture inventing an artificial man out of nothing, with no theory of computation and no theory of much else, gave him something to lose. They could have left him invulnerable. He was made of bronze; it would have cost them nothing. Instead they gave him a vein, and a pin, and a way to be emptied. Some intuition told them that a thing which patrols an island, and chooses, and defends, has to be the kind of thing that can be drained.

I want to hold that intuition up against the most confident claim of our own moment, because I think the Bronze Age had hold of something we have mislaid.

On the twenty-third of July this year Elon Musk sat down with Zanny Minton Beddoes of The Economist and said this. Artificial intelligence “may exceed the sum of human intelligence in around five years”, and in ten will be far greater than it. Married to large fleets of robots, artificial intelligence will produce what he calls a quasi-infinite economy, one in which “money won’t matter in 2036”, because money is a claim on scarce things and nothing much will be scarce. He does not expect humans to remain in charge. He offered the comparison himself: the gap will be something like the one between a human and a chimpanzee, and it is hard to imagine the chimpanzee running the place. He has previously put the odds of the whole business ending us at ten or twenty per cent. Asked what followed from that: “Let’s enjoy the ride is my philosophy at this point.”

It would be easy, and wrong, to treat that as one man’s eccentricity. Musk is not the author of this view but its clearest exponent, which is a different and more useful thing: he says out loud, and in order, what a great deal of the industry believes and phrases more carefully. The view is that intelligence is a quantity. It is produced by computation. More computation, better arranged, produces more of it. The binding constraint at the moment, he says, is “actually slightly more on power and cooling than it is on AI chips”, and constraints of that kind are engineering problems, and engineering problems yield.

You can see exactly where that comes from, which is to its credit. A rocket is an optimisation problem. A battery is an optimisation problem. A factory is an enormous optimisation problem with a schedule attached. In every one of those domains, minimising the error is the intelligent thing to do, and whoever minimises it fastest wins. A man who has spent thirty years watching that hold has earned the generalisation. The question is whether it survives the trip into a different category.

Four weeks later the same interviewer put a version of the question to Yuval Noah Harari, who states he agrees with Musk except for the inevitability. He thinks a machine that outruns us is a choice being made rather than a weather front coming in. But listen to what he grants along the way. He grants the abundance, and agrees money may stop meaning much. He grants agency without consciousness: a system does not have to feel anything to pursue a goal, and goal-seeking is enough to make it an actor. And then he asks the question that ought to have stopped the room. “What is accountability for an entity that cannot suffer?” A chief executive who cannot be jailed, cannot be shamed and cannot be ruined has escaped every piece of machinery we built to restrain chief executives.

It is the right question, and he leaves it standing. His answer is oversight: keep a human in the arrangement and accountability returns through that person. Which is Musk’s answer too, arrived at from the opposite direction. One of them says the machines are coming and we should enjoy the ride. The other says they are coming only if we allow it and we should refuse. Asked who carries the loss, both point at a human being.

Here is the older claim they both run into.

Thermodynamics is the strangest item in the physics curriculum, because it is the only part of it that is not a theory of what things are. It came out of engineering (Carnot, in 1824, counting what a steam engine could and could not be made to do) and it makes no commitment whatever about atoms, fields or spacetime. It only constrains. And because it says nothing about the furniture of the world, every new account of that furniture has had to walk past it on the way in. Relativity left it standing. Quantum mechanics left it standing. Black holes looked for a while like the exception the field had been waiting for, until Bekenstein and Hawking worked out that a black hole carries an entropy of its own and the books balanced after all. Small systems can run backwards briefly, and the fluctuation theorems say exactly how much less often that happens as a system grows; the law falls out of them as the limit. Eddington’s verdict in 1928 was that a theory found to be against the second law has “nothing for it but to collapse in deepest humiliation”. Einstein, who overturned a fair amount himself, thought classical thermodynamics the one physical theory that would never be overthrown. Two centuries, every revolution physics has had, and no exception yet found.

But there are two versions of that law running together in this argument, and everything turns on the difference between them.

The first is the version an engineer meets. Work costs. Order costs. The heat has to go somewhere, and somebody pays the bill. Call it the infrastructure version. It applies to a steam engine, a brain, a data centre, and there is nothing mysterious about it.

The second is the one Schrödinger reached for in Dublin in 1943, lecturing on a biological question: why does living matter conspicuously fail to run down the way everything else does? His answer was that an organism survives by continually drawing order out of its surroundings: it feeds, as he put it, upon negative entropy. Ilya Prigogine spent the following decades putting mathematics under it, and took a Nobel for the theory of dissipative structures: systems held far from equilibrium that keep their order indefinitely, but only by dissipating continuously, never by arriving. Call this the existential version. It says something stronger than that order costs. It says that for such a structure the payment is conditional on its own conduct, and that if it stops paying there is no structure left to have been wrong.

That second version is where everything we care about comes from. Because the organism must keep buying its own continuation, the world stops being uniform for it. A gradient becomes food. A temperature becomes danger. Something that was merely there becomes something that helps or something that harms. That division, in a bacterium with no nervous system whatever, is the first appearance in the universe of anything mattering to anything. Not brains, not language, not symbols: the requirement to keep paying. What we later call meaning is that same division, refined by a receiver with enough structure to carry a history.

Which is also why meaning has an arrow and information does not. Shannon’s measure is symmetric: the information in a channel is the same seen from either end, and entirely indifferent to what happens next. He said so himself on his first page: “These semantic aspects of communication are irrelevant to the engineering problem.” He was not overlooking meaning; he was setting it aside, because it was not what he was measuring. Meaning is what a message does to a receiver that has a stake in what it does next. Information is a state function. Meaning is a path function.

The most famous attempt on the law, as it happens, was also an argument about intelligence. In 1867 James Clerk Maxwell imagined a small being at a shutter between two chambers of gas, letting the fast molecules through one way and the slow ones the other, building a temperature difference out of nothing but discernment. It took the better part of a century to kill the demon, and the manner of its death is the point. Szilard saw that the measurement must cost something; Landauer worked out in 1961 that the cost is not in the looking but in the forgetting, since erasing a bit dissipates a minimum quantity of heat; Bennett closed the case in 1982. The demon can sort for nothing. What it cannot do is keep sorting, because its memory fills, and clearing the record is where the bill falls due. It runs against the gradient exactly as long as it can afford to. Then it pays. And that is not a defeat for the demon, it is a description of every living thing that has ever existed. A cell is a demon that pays continuously. Your own share of the arrangement runs at about twenty watts and stops within minutes if the supply is cut.

Now put the machine against both versions, because this is where the forecast comes apart.

On the infrastructure version, Musk is not merely compliant, he is impressive. A data centre is a furnace with a car park; Landauer’s bound applies to silicon before it applies to anything else; and the binding constraint is power and heat, exactly as he says. His answer is to put the compute in orbit: “once we address the power constraint with AI data centers in space, then the constraint will once again be chips.” That is a thermodynamically literate reply. Orbit is the one address where the cooling is free, because the heat has somewhere to go and nobody invoices you for the radiator. He is not ignoring the second law; he is engineering around its infrastructure version at planetary scale, and he may well succeed.

The existential version does not appear in the account at all. Every constraint he names (chips, power, cooling) belongs to the first kind. It is not that the machines are failing the second test; it is that nobody has put it on the list. Nothing about the model’s continuation is conditional on its performance. If it is wrong about everything for a week, the same power flows, the same weights persist, the same service answers. The cost is imposed on its behalf rather than incurred on its own, and a reward signal does not fix this, because the reward is issued from outside as well. A stake on loan is not a stake. More chips and more electricity settle the infrastructure account in full. They do not open an existential one, and there is no quantity of either that would.

The same borrowed stake runs through every machine that appears to own one. A washing machine senses its load. A guided missile hunts. A driverless car strains not to wreck itself. But behind each of them stands a supervisor who holds the account and settles the entropy bill: a maker, an operator, an owner. The stake is real, and it is never the machine’s. That is why none of them breaks the second law: it never pays, because someone else does. Which leaves the question the forecast never reaches. Who supervises the superintelligence, and holds what it has to lose? On the plan as announced, the supervisor is the man announcing it. Ahem.

Harari’s question comes back here, and the answer to it is older than the question. He has said that consciousness is suffering, a formulation he takes from a contemplative tradition rather than a laboratory. Set aside whether it is the whole of consciousness. Take the smaller claim, that suffering is what it is like to be losing, and ask what has to be true of a thing before it can lose.

It has to be paying. A structure held far from equilibrium is buying every moment of its own persistence, and the purchase can fail. That is the condition under which some states of the world become worse for a thing than others, and that division is the raw material from which suffering is later built. Nothing that is not paying can be losing. Nothing that cannot lose has any use for a signal that says this is going badly.

Biology reached the same place from its own end. Antonio Damasio has spent a career arguing that feelings are the mind’s reading of the body’s housekeeping, that they exist because the body can fail, and with Kingson Man he proposed the engineering version outright: if you want a machine that feels, build one that is genuinely vulnerable and must keep itself alive. Mark Solms puts affect at the root of consciousness rather than its surface, for the same reason. They arrive by way of homeostasis. This route arrives one floor down, at the constraint that makes homeostasis compulsory.

There is a second and quieter difference. The demon works in the present tense, because the molecule arrives now. A trained model is a completed integral, frozen at a date: it can read the meaning in what you send it, often better than most people would, but it cannot be changed by the reading. Give it a memory file and it accumulates files, not itself. And what it accumulates from is text, increasingly images and video, but still the cast shadow rather than the thing. Michael Polanyi’s point about tacit knowledge was that we know more than we can tell: the surgeon’s hands, the feel of a hull going soft, what a room is doing before anyone in it has spoken. The model is working from the reviews rather than the music.

Even Ray Kurzweil, who has spent thirty years insisting the growth is exponential and still puts human-level machine intelligence at 2029, notes that computational capacity alone will not create artificial intelligence. What he thinks is missing is an understanding of the brain, to be got by imaging it and reverse-engineering what is found. Which quietly concedes that the biological arrangement has something the scaling does not; he simply locates it in the wiring diagram. It is not in the wiring. It is in the fact that the wiring has to be paid for, continuously, or it stops.

There is a practical edge on this, and it is the one worth being nervous about. Prediction is cheap to undo: a wrong answer costs a retry. Action is not. And the deployment is running hard at exactly the irreversible half: money moved, code shipped, contracts signed, physical machinery pointed at the world with a schedule of its own. We are coupling a system with no exposure to consequences that cannot be taken back. That is not the familiar story about a machine that wants the wrong thing but a plainer arrangement, and a worse one: an actor operating in the one direction time runs, with nothing whatever riding on how it turns out.

Which is where the ride comes in. Helios had a son, Phaethon, who asked for the sun chariot for a day. His father was bound by an oath and could not refuse him. The horses felt the lighter hand at once, the chariot went too high and then too low, the earth burned in bands, and it ended with a thunderbolt and a boy in a river. Phaethon’s failing was not that he doubted the chariot; he loved the chariot. It was that he wanted the ride, and the ride is not the same thing as the hands. Enjoy the ride is a strange sentence from Musk of all people, the man who put the extinction number into the public record. One chance in five, or one in ten, is not a ride. It is a wager laid over a city, at odds he named himself, by a man who has spent a decade shortening them.

So where does the burden sit? Nothing alive has ever escaped the existential version of the law, and nothing we have built has ever entered it. Not one machine, in two centuries of building them, has had its own continuation ride on getting anything right. That is a long unbroken record, and a record of that kind does not prove anything, but it does decide who has to do the work. The claim that a system can take the infrastructure half and skip the existential half, and still hold preferences and stakes and a future worth protecting, is the claim that needs testing. It is the exception rather than the default, and it is being asserted rather than demonstrated.

I should say what would show me wrong, because a burden-shift is only honest if my own side can lose. Build a system whose continuation is genuinely conditional on its own conduct: nobody restoring it, nobody supplying the reward, being wrong shortening its life. Then watch whether the behaviours turn up. Does it economise when it would be easy to be lavish? Does it refuse a task whose cost to itself exceeds what it stands to gain? Does it treat one of its own futures as worth more than another without having been handed the exchange rate? Nothing in physics forbids building that, and a fleet of robots is one plausible road towards it. If it were done and those behaviours appeared, this essay would apply to the result rather than deny it.

The harder objection is that the thermodynamic account explains how our kind of intelligence arose without describing what intelligence has to be. Evolution built us under an energy constraint, and stakes are what that constraint felt like from the inside; scaffolding, and scaffolding comes down. I do not think that is settled, and I would not pretend otherwise. But notice what the concession costs. It grants that meaning on the machine side would be something newly constructed rather than something already present and merely small, a thing to be added, not a thing to be scaled into. And that is not the claim being made in public. The claim being made in public is that more compute, better arranged, arrives there on its own.

So, back to Crete.

What is striking about Talos is not that he was beaten but what his makers had to give him first, so that beating him made sense as a story. They gave him a vein, and a nail, and something that could be let out of him and not put back. He was a machine and he was inside the arrangement: he could be emptied, and because he could be emptied he had something to defend, and because he had something to defend the story works at all. The storytellers put the vein in without being able to say why.

Harari’s own proposal, when it came, was the mirror image of the Cretans’. He would ban the mimicry. No training a machine to pass as a person, no granting it the standing of a thing that can be wronged. Talk to it about your health if you want, so long as nobody in the room forgets which of you can be harmed. The storytellers gave their machine a vein so the story would work. He would forbid the machine even the appearance of one, so that we do not mistake where the vein actually is. The moves run opposite and rest on the same conviction, that everything follows from who can be emptied.

Musk and the industry are building the bronze, and it is very good bronze, better every year, and the people who keep announcing the plateau have been wrong for a decade. What is not clear, and what nobody has yet made them answer, is whether the thing they are describing is meant to sit inside the arrangement or outside it.

Suppose it complies. Then its persistence has to be conditional on its own conduct, and somebody has to build it a vein: a metabolism it has to earn, a body that runs down, a future that its own behaviour either secures or loses. That is not what is being built, and it is not what more compute delivers. On that path the abundance may well arrive, and the superintelligence with it, and they arrive as an instrument of extraordinary power in which nothing is at stake, the meaning staying on our side of the glass, because we remain the only things in the room that can be harmed.

Or suppose it does not comply. Then something will hold significance, and preference, and a future worth protecting, while paying nothing and risking nothing. That is not a faster chip or a larger cluster or a better arrangement of attention heads. That is the first exception to the second law of thermodynamics in two hundred years, found not in a laboratory but in a product cycle. It would be the largest result in the history of physics, and it is being announced as a delivery date.

So what becomes of a stake the machines cannot hold? It does not disappear. It stays where it has always been, with the things that can be drained, which is to say with us. The only live question is whether it is held by one of us or by all of us.

Every machine we have built so far has had a supervisor, somebody the bill could be sent to. That arrangement works while the machine is a washing machine. It fails at the scale being described, because no individual can be exposed in proportion to a one-in-five chance of losing everything. There is no person alive who can be harmed that much, and a stake that cannot be felt is not a stake at all.

Which is why the answer is not a better supervisor. It is the arrangement life has already built for exactly this problem: courts, parliaments, insurers, professions, the slow sediment of rules laid down by people who could be harmed and knew it. Those institutions are clumsy and they are always late, and they are the only holders of a stake that scale, because they are made of the only material that has ever carried one.

Harari, asked the same question from the other end, arrives in the same place. His answer is democracy, and his reason for it is structural rather than sentimental: democracies have moving parts, and they spread power across people who can be turned out. He gets there by watching what happens to accountability when the actor cannot be punished, which is this road walked without the physics under it. It is worth something that two accounts with almost nothing else in common set the answer down in the same building.

Which leaves the question that was never put to him. The AI that Elon Musk is describing: does it comply with the second law of thermodynamics, or doesn’t it? It admits of only two answers, and both of them are a great deal more interesting than five years.

Note on method. The ideas here are mine, and have been the subject of a human quest you can follow at www.integraloflife.com. As an engineer I am comfortable having them written with an LLM in support, as much as you presumably are about the software that helped design the aircraft you fly in.