P(doom)
In short
P(doom) is a stated probability that artificial intelligence causes an existentially catastrophic outcome. The disaster, the time span and the conditions differ between uses, so two people's numbers can answer different questions.
- Also called
- p(doom), Probability of doom
- Topic
- Safety, ethics and risk
- Level
- Some background helps
Wikipedia describes P(doom), in AI safety, as the probability of an existentially catastrophic outcome from artificial intelligence. It says the exact outcome differs from one prediction to another, and that the figures generally point at existential risk from artificial general intelligence. The article says the phrase began as shorthand in the rationalist community and among AI researchers, and became prominent in 20231.
Isaac King argues that the shorthand still hides the question. He lists uses that include AI killing people within 5 to 10 years, within 80 to 200 years, death soon after artificial general intelligence, death soon after superintelligence, and sometimes risks that are not from AI at all. On that account, two stated probabilities can disagree because they are about different events2.
A 2023 survey of 2,778 researchers who had published in top-tier AI venues shows how the wording moves the number. One question assumed high-level machine intelligence would exist, then asked about its long-run impact. The median probability placed on an "extremely bad" outcome, "e.g. human extinction", was 5%, and the mean was 9%3.
Three further questions asked about human extinction or similarly permanent and severe disempowerment. The medians were 5% for future AI advances causing that outcome, 10% when the question named an inability to control advanced AI, and 5% when the outcome was limited to the next 100 years. For that 100-year question, 655 respondents gave a mean of 14.4%. The paper says these are expert forecasts and should not be read as a reliable guide to objective truth3.
Example
In the 2023 survey, 10% was the median answer when researchers were asked about human extinction or similarly severe disempowerment caused by an inability to control advanced AI. The median was 5% on two other wordings of that outcome. Those figures are what respondents said, not a score measured on a model3.
Why it matters
The phrase lets people put a number on a fear of extreme AI outcomes. King argues that the number is easy to misunderstand because the disaster and the time span are often left unstated, and the survey medians move from 5% to 10% when the question changes23.
Questions people ask
What does p(doom) mean in AI?
Wikipedia defines it as the probability of an existentially catastrophic outcome from artificial intelligence, and notes that the exact outcome differs by prediction1. King adds that speakers also differ on the time span and on whether the probability is conditional on a particular kind of AI2.
Is p(doom) a scientific measurement?
The 2023 survey records probabilities that publishing AI researchers stated. Its authors say those predictions should not be treated as a reliable guide to objective truth, and the survey does not measure a property of a model3.
What p(doom) did AI researchers give in the 2023 survey?
The survey did not use the name p(doom). On three questions about human extinction or similarly permanent and severe disempowerment, the medians were 5%, 10% and 5%. The 100-year version had a mean of 14.4%. A separate long-run question, assuming high-level machine intelligence exists, had a median of 5% for an extremely bad outcome such as extinction3.