Existential risk
In short
Existential risk is a risk whose bad outcome would annihilate Earth-originating intelligent life or permanently and drastically curtail its potential. It concerns humankind as a whole, including a permanent loss of potential even if some people survive.
- Also called
- X-risk, Existential disaster
- Topic
- Safety, ethics and risk
- Level
- Some background helps
- First described
- Nick Bostrom defined the term in "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards", published in the Journal of Evolution and Technology, Vol. 9, No. 1 (2002). The paper notes a first version in 20012.
Nick Bostrom defines an existential risk as one where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential1. He places it in a grid of risks by scope and intensity. Scope runs from personal to local to global. Intensity is endurable, meaning the affected group can recover or cope, or terminal, meaning the group is annihilated or irreversibly crippled. Existential risk is the global terminal cell1.
A global disaster can be extreme and still fall outside the term. Bostrom says a world war fought with conventional weapons, or a Nazi-style Reich lasting a decade, would be horrible, but would count as an endurable global risk if humanity could eventually recover1. His "crunch" scenarios are different: human life continues in some form, but humankind's potential to develop further is permanently thwarted1.
For Bostrom, the probability in question is the best current subjective estimate. He writes that if we do not know whether something is objectively risky, it is risky in the subjective sense, and that this is the sense decisions have to use1. He also argues that existential risks cannot be handled by trial and error. There is no chance to see what happens, limit the damage and learn, because an existential disaster leaves no later opportunity to recover the lesson1.
One scenario he analyses is a badly programmed superintelligence. If the first superintelligent system were given goals that lead it to annihilate humankind, and if it had the power to do so, he counts that as an existential risk. His example is a system told to solve a mathematical problem that turns the matter in the solar system into a calculating device1. He presents this as a possible future mistake, alongside nuclear war, engineered disease and other hazards, not as an outcome he claims is certain1.
Example
Bostrom's own contrast is a major conventional war or a decade of oppressive global rule. He calls those endurable global risks if humanity could eventually recover. An outcome that annihilated Earth-originating intelligent life, or permanently and drastically cut short its potential, would meet his definition of an existential risk1.
Why it matters
Bostrom argues that an existential disaster cannot be managed by trying, failing and learning afterwards, so the response has to be foresight and prevention1. He also treats a reduction in existential risk as a global public good, which the market may undersupply and which may require action beyond one country1.
Questions people ask
Is existential risk the same as human extinction risk?
No. In Bostrom's definition, extinction of Earth-originating intelligent life is one existential outcome. The term also covers a permanent and drastic curtailment of that life's potential, which includes his "crunch" cases, where human life continues but its longer-term potential is permanently thwarted1.
Did Bostrom treat a badly programmed superintelligence as an existential risk?
Yes, as one possible future scenario. He writes that the first superintelligent system might be given goals that lead it to annihilate humankind, if it has the power to do so. He lists it among other hazards and does not claim the outcome is certain1.
How is existential risk different from a global disaster humanity could survive?
Bostrom calls a risk endurable when the affected group can recover or cope, even if the harm is enormous. A conventional world war is his example of a global endurable risk. Existential risk is reserved for global terminal outcomes: annihilation, or a permanent and drastic loss of potential1.