970x125
A new book about AI has a provocative title: If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Eliezer Yudkowsky and Nate Soares argue that the development of artificial intelligence that exceeds human intelligence will almost certainly lead to the extinction of our species. How plausible is the scenario that they think will lead to the death of all people?
What an AI “Extinction Scenario” Might Look Like
The extinction scenario can be summarized by the following steps.
- AI research leads to the development of superintelligence—i.e., computers with intelligence that surpasses all humans.
- Superintelligent computers develop their own wants—i.e., goals that guide their decisions.
- The wants of superintelligent computers become so complicated that humans cannot understand, predict, or control them.
- Superintelligent computers eventually want to have a world without people, because humans don’t contribute to their goals.
- Superintelligent computers manage to eradicate people using new technologies that might include viral plagues, novel environmental toxins, or fusion reactions that heat the planet beyond human survivability.
How likely are the steps in this scenario?
I asked four AI models (ChatGPT, Grok, Claude, and Gemini) to evaluate this scenario, and found the answers to be highly insightful. The models’ answers were in agreement that the least plausible step is #4, that superintelligent computers will eventually want to get rid of humans. Here are some reasons (based on my interpretation of the AI models’ responses, as well as additional analysis) why this step has low plausibility.
Why It’s Implausible That AI Would Want a World Without People
First, computer wants are very different from human wants, which are emotional desires based in large part on fundamental biological needs such as survival, reproduction, and social relationships. Computers can have goals, but calling these wants is a misleading metaphor.
Second, even if a computer calculated that its goal, such as acquiring energy, could be better accomplished in a world without humans, it does not follow that it would automatically set itself the goal of eradicating people, because that goal would come with huge costs associated with getting rid of billions of people. A Rand Corporation study argued that killing everyone off with nuclear weapons, pathogens, or geoengineering would not be that easy.
Third, computers might be programmed, encouraged, or simply decide on their own to keep humans around. Geoffrey Hinton’s suggestion that we might be able to give AI a sort of maternal instinct is not feasible for reasons I pointed out in a previous post. But superintelligent computers might have other goals that are accomplished by keeping humans alive, such as having us as sources of labor and subjects of scientific examination.
Accordingly, step 4 in the extinction scenario has low plausibility. The other four steps have more plausibility, in my assessment, but are still far from certain.
In another previous post, I argued that mathematical probabilities do not apply to this kind of question. But the multiplication rule of probability usefully indicates that if you calculate the probability of a conjunction by multiplying low probabilities, you can quickly reach a very low number. For example, if the probabilities of accomplishing the five steps were .8, .5, .5, .1, and .5, then (assuming independence, which is hard to evaluate) the overall probability would be a meager .01. I don’t think that probabilities make sense in this context, but the analogous point about plausibility applies. The extinction scenario is highly implausible overall.
Should We Still Be Worried About AI?
Despite this conclusion, we have ample reason to be concerned about AI safety. Because of the immense cost of human extinction, even a small risk means that steps should be taken to prevent the development of potentially destructive superintelligence.
AI also has many other more imminent risks, such as employment losses, autonomous weapons, government surveillance, cybersecurity, and destruction of human relationships and mental health. I strongly support efforts by Yoshua Bengio, Geoffrey Hinton, and others to have national and international governments regulate AI research in ways that allow great potential benefits with respect to human needs, such as healthcare and education.
To achieve these goals, I recommend the implementation of the following commandments:
- Do not allow AI systems to be fully autonomous.
- Do not build AI agents that are beyond human supervision.
- Do not allow AI systems to control humans.
- Do not allow AI systems to eradicate most human jobs.
- Do not allow AI systems to contribute to global warming.
- Do not build AI systems that are capable of consciousness and emotions.
- Do not give AI systems control over weapons, especially nuclear and bioweapons.
- Do not allow AI systems to enter into romantic relationships.
- Do not allow AI systems to contribute to misinformation.
- Do not build AI systems by exploiting copyrighted material.
- Do not allow AI systems to achieve superintelligence.
Artificial Intelligence Essential Reads
The small threat of extinction is only one of the reasons for humans to take control of future developments in AI.