Loading
What changed between revision 1 and revision 2, compared over the article's markdown source with its frontmatter — so a change to status, evidence or reversibility shows up as the edit it is.
Older · revision 1
Import· initial corpus import, not an edit
Initial import of content/existential-risk.md — the filesystem corpus, unchanged. Not an edit.
13.2 kB
Revision 1
Revision 2
lines 7–13 → 7–136 unchanged lines not shown
categories: ["society", "foundations"]categories: ["society", "foundations"]tags: ["risk", "governance", "biosecurity", "longtermism", "forecasting", "policy"]tags: ["risk", "governance", "biosecurity", "longtermism", "forecasting", "policy"]summary: "The category of risks that would end humanity or permanently destroy its potential, and the research programme built around identifying and reducing them."summary: "The category of risks that would end humanity or permanently destroy its potential, and the research programme built around identifying and reducing them."updated: "2026-07-27"updated: "2026-10-29"humanEvidence: "No existential catastrophe has occurred, so the anthropogenic probabilities are stated credences rather than estimates from data; only the natural background rate is bounded, by the fossil and geological record."humanEvidence: "No existential catastrophe has occurred, so the anthropogenic probabilities are stated credences rather than estimates from data; only the natural background rate is bounded, by the fossil and geological record."reversibility: "irreversible"reversibility: "irreversible"issues: ["The 2022 asteroid deflection result is stated without a citation."]issues: ["The 2022 asteroid deflection result is stated without a citation."]lines 27–70 → 27–7013 unchanged lines not shown
}}`````` **Existential risk** is a risk of an outcome that would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential. The definition is [[nick-bostrom]]'s, from a 2002 paper that gave the field its name, and its distinguishing feature is not severity but irreversibility.[^bostrom2002] A catastrophe that kills a large fraction of humanity and from which the survivors recover is, in this framework, categorically different from one that forecloses the future — however similar the two look in the immediate aftermath.**Existential risk** is a risk of an outcome that would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential. The definition is [[nick-bostrom]]'s, from a 2002 paper that gave the field its name, and what distinguishes the category is irreversibility rather than severity.[^bostrom2002] In this framework a catastrophe that kills a large fraction of humanity, but from which the survivors recover, is categorically different from one that forecloses the future, however similar the two look in the immediate aftermath. ## The definition and its taxonomy## The definition and its taxonomy Bostrom's original classification sorted outcomes by how the potential is lost. **Bangs** are sudden extinction events. **Crunches** leave humanity alive but permanently unable to develop further. **Shrieks** reach a form of posthuman existence realising only a fraction of what was possible. **Whimpers** are gradual erosions. A later restatement replaced the taxonomy with two axes — scope and severity — with existential risk in the far corner of both.[^bostrom2013]Bostrom's original classification sorted outcomes by how the potential is lost. **Bangs** are sudden extinction events. **Crunches** leave humanity alive but permanently unable to develop further. **Shrieks** reach a form of posthuman existence realising only a fraction of what was possible. **Whimpers** are gradual erosions. A later restatement replaced the taxonomy with two axes, scope and severity, and placed existential risk in the far corner of both.[^bostrom2013] Treating a permanent curtailment of potential as equivalent to extinction requires the view that the value at stake is mostly in the future rather than the present, the premise connecting this research to longtermism and one many critics reject. The related term *global catastrophic risk* is broader and does not require irreversibility.Treating a permanent curtailment of potential as equivalent to extinction requires the view that the value at stake lies mostly in the future rather than the present. That premise connects this research to longtermism, and many critics reject it. The related term *global catastrophic risk* is broader and does not require irreversibility. ## The mechanism## The mechanism What makes a risk existential rather than merely catastrophic is the absence of a recovery path. Three properties tend to produce that.A risk is existential rather than merely catastrophic when it leaves no recovery path. Three properties tend to remove that path. **Self-propagation.** An engineered pathogen, unlike a bomb, continues to act after release. The same is true of the self-spreading edits described in [[gene-drive]] and of the chirally inverted organisms discussed in [[mirror-life]], where the specific concern is that immune systems evolved against ordinary chirality might not recognise the invader at all. The original self-replication scenario, [[grey-goo]], turned out to rest on engineering assumptions its own author later withdrew — a case worth remembering when assessing the current ones.**Self-propagation.** An engineered pathogen, unlike a bomb, continues to act after release. The same is true of the self-spreading edits described in [[gene-drive]] and of the chirally inverted organisms discussed in [[mirror-life]], where the specific concern is that immune systems evolved against ordinary chirality might not recognise the invader at all. The original self-replication scenario, [[grey-goo]], turned out to rest on engineering assumptions its own author later withdrew, a precedent relevant to assessing the current scenarios. **Loss of the capacity to respond.** A sufficiently capable optimising system pursuing objectives that diverge from human ones need not be malicious to be unrecoverable; it need only be better at acting in the world than the people trying to stop it. This is the argument treated in [[artificial-general-intelligence]] and [[technological-singularity]], and it is where the field's attention has concentrated since 2020.**Loss of the capacity to respond.** A sufficiently capable optimising system pursuing objectives that diverge from human ones could be unrecoverable without being malicious, provided it is better at acting in the world than the people trying to stop it. This is the argument treated in [[artificial-general-intelligence]] and [[technological-singularity]], and it is where the field's attention has concentrated since 2020. **Lock-in.** A stable global order that permanently prevents change — through surveillance, through control of the means of coordination, or through value entrenchment — produces a crunch without killing anyone. This branch receives the least analysis, partly because it is hard to distinguish in advance from ordinary political failure. It is also where the field's concerns intersect with the trajectories set out in [[future-of-humanity]] and with the question, raised in [[machine-consciousness]], of whether the entities inheriting such an order would be moral patients at all.**Lock-in.** A stable global order that permanently prevents change, whether through surveillance, control of the means of coordination, or value entrenchment, produces a crunch without killing anyone. This branch receives the least analysis, partly because it is hard to distinguish in advance from ordinary political failure. It is also where the field's concerns intersect with the trajectories set out in [[future-of-humanity]] and with the question, raised in [[machine-consciousness]], of whether the entities inheriting such an order would be moral patients at all. Extinction from a single physical event is comparatively hard: humans occupy every continent, and the record includes survived population bottlenecks.Extinction from a single physical event is comparatively unlikely, because humans occupy every continent and the record includes population bottlenecks that the species survived. ## Plausibility## Plausibility The most-cited quantitative estimates are Toby Ord's, published in 2020 and presented explicitly as his own subjective credences rather than as calculations.[^ord2020] He puts the probability of an existential catastrophe in the twenty-first century at roughly one in six; unaligned artificial intelligence at roughly one in ten; engineered pandemics at roughly one in thirty; and all natural risks combined at something like one in ten thousand.The most-cited quantitative estimates are Toby Ord's, published in 2020 and presented explicitly as his own subjective credences rather than as calculations.[^ord2020] He puts the probability of an existential catastrophe in the twenty-first century at roughly one in six; unaligned artificial intelligence at roughly one in ten; engineered pandemics at roughly one in thirty; and all natural risks combined at something like one in ten thousand. The natural-risk figure is the only one with a defensible derivation. A species facing a high per-century probability of extinction from asteroids, supervolcanism, or naturally arising disease would be unlikely to persist for hundreds of thousands of years — and mammalian species typically endure on the order of a million. The fossil and geological record bounds the natural rate from above, and the bound is low.The natural-risk figure is the only one with a defensible derivation. A species facing a high per-century probability of extinction from asteroids, supervolcanism, or naturally arising disease would be unlikely to persist for hundreds of thousands of years, and mammalian species typically endure on the order of a million. The fossil and geological record bounds the natural rate from above, and the bound is low. No comparable derivation exists for the anthropogenic figures, and this is the field's central methodological problem. There is no base rate for technologies never deployed, no frequency data, and no way to calibrate a forecaster on events that happen at most once. Estimates correlate strongly with the estimator's prior commitments. As numbers they carry a precision the reasoning does not support; as ordinal rankings — anthropogenic risks dominate natural ones, and biology and AI dominate within them — they command much broader agreement.No comparable derivation exists for the anthropogenic figures, which is the field's central methodological problem. There is no base rate for technologies never deployed, no frequency data, and no way to calibrate a forecaster on events that happen at most once. Estimates correlate strongly with the estimator's prior commitments. As numbers they carry a precision the reasoning does not support. As ordinal rankings they command much broader agreement: anthropogenic risks dominate natural ones, and biology and AI dominate within the anthropogenic group. > [!caution] What the probabilities are> [!caution] What the probabilities are> Ord's figures are stated credences from one careful analyst, not scientific estimates. They are frequently quoted without that qualification. Their most defensible use is comparative: they say which risks their author thinks deserve the most attention, not how likely the world is to end.> Ord's figures are stated credences from one careful analyst rather than scientific estimates, and they are frequently quoted without that qualification. Their most defensible use is comparative: they say which risks their author thinks deserve the most attention, not how likely the world is to end. The biological case is the one where the reasoning is most concrete, because the enabling capabilities are documented rather than projected. [[dual-use-research]] traces the specific experiments — enhanced transmissibility, viral synthesis from published sequence, evasion of countermeasures — that turn a general worry into a mechanism, and the whole-genome construction methods in [[synthetic-genomes]] supply the assembly step. The design tools described in [[ai-protein-design]] raise a further question: whether screening of synthesised sequences can recognise a protein that was designed rather than copied. No comparable chain of demonstrated capabilities exists for any other candidate risk.The biological case is the one where the reasoning is most concrete, because the enabling capabilities are documented rather than projected. [[dual-use-research]] traces the specific experiments (enhanced transmissibility, viral synthesis from published sequence, evasion of countermeasures) that turn a general worry into a mechanism, and the whole-genome construction methods in [[synthetic-genomes]] supply the assembly step. The design tools described in [[ai-protein-design]] raise a further question: whether screening of synthesised sequences can recognise a protein that was designed rather than copied. No comparable chain of demonstrated capabilities exists for any other candidate risk. ## The vulnerable world hypothesis## The vulnerable world hypothesis Bostrom's sharpest contribution to policy is a thought experiment.[^bostrom2019] Imagine technological discovery as drawing balls from an urn: most are white and beneficial, some grey and mixed. The hypothesis asks what happens if the urn contains a black ball — a technology that by default destroys the civilisation that discovers it. His illustrative case is a method for building nuclear weapons from readily available materials, which would make mass destruction available to anyone sufficiently motivated.Bostrom's sharpest contribution to policy is a thought experiment.[^bostrom2019] It pictures technological discovery as drawing balls from an urn, in which most balls are white and beneficial and some are grey and mixed. The hypothesis asks what happens if the urn contains a black ball: a technology that by default destroys the civilisation that discovers it. His illustrative case is a method for building nuclear weapons from readily available materials, which would make mass destruction available to anyone sufficiently motivated. The argument's force is in what follows. If such a technology exists and is drawn, only four responses are available: stop drawing, ensure no actor has the motive, achieve preventive policing effective enough to intercept every attempt, or achieve global governance capable of coordinating a halt. Bostrom notes without enthusiasm that the last two require surveillance and centralisation most people would regard as intolerable, and that the first two are not achievable. The paper is best read as a demonstration that a certain class of world admits no acceptable solution, and therefore as an argument for finding out early whether this is such a world.The argument's force lies in what follows. If such a technology exists and is drawn, only four responses are available: stop drawing, ensure no actor has the motive, achieve preventive policing effective enough to intercept every attempt, or achieve global governance capable of coordinating a halt. Bostrom notes without enthusiasm that the last two require surveillance and centralisation most people would regard as intolerable, and that the first two are not achievable. The paper is best read as a demonstration that a certain class of world admits no acceptable solution, and therefore as an argument for finding out early whether this is such a world. ## Criticism## Criticism lines 72–92 → 72–921 unchanged line not shown
**Pascal's mugging.** Multiplying a very small probability by an astronomically large value can make any intervention appear to dominate, a decision-theoretic pathology that Bostrom himself named. Any framework valuing the whole future is vulnerable to it, and no accepted resolution exists.**Pascal's mugging.** Multiplying a very small probability by an astronomically large value can make any intervention appear to dominate, a decision-theoretic pathology that Bostrom himself named. Any framework valuing the whole future is vulnerable to it, and no accepted resolution exists. **The values are contested.** Émile Torres and Timnit Gebru have argued that the cluster of ideas linking existential risk, longtermism, and [[transhumanism]] rests on assumptions about what counts as valuable — vast future populations, expansion beyond Earth, digital minds — that are neither neutral nor widely shared, and that the framework licenses discounting present harms against speculative future ones.[^torres2024] Defenders reply that reducing extinction risk is supported by almost any value system and does not require the longtermist premises.**The values are contested.** Émile Torres and Timnit Gebru have argued that the cluster of ideas linking existential risk, longtermism, and [[transhumanism]] rests on assumptions about what counts as valuable (vast future populations, expansion beyond Earth, digital minds) that are neither neutral nor widely shared, and that the framework licenses discounting present harms against speculative future ones.[^torres2024] Defenders reply that reducing extinction risk is supported by almost any value system and does not require the longtermist premises. **Institutional credibility.** The field's funding and reputation were damaged by the 2022 collapse of a major donor in the effective-altruism ecosystem, and Oxford's Future of Humanity Institute — its original home — was closed by the university in 2024 after protracted administrative disputes. Work continues at Cambridge's Centre for the Study of Existential Risk, at the Future of Life Institute, and at biosecurity centres, but the centre of gravity has moved toward AI policy organisations with narrower remits.**Institutional credibility.** The field's funding and reputation were damaged by the 2022 collapse of a major donor in the effective-altruism ecosystem, and Oxford's Future of Humanity Institute, its original home, was closed by the university in 2024 after protracted administrative disputes. Work continues at Cambridge's Centre for the Study of Existential Risk, at the Future of Life Institute, and at biosecurity centres, but the centre of gravity has moved toward AI policy organisations with narrower remits. ## Mitigation and governance## Mitigation and governance The field's characteristic policy proposal is [[differential-technological-development]]: rather than accelerating or halting technology in general, deliberately advance defensive and stabilising capabilities ahead of destabilising ones. Applied to biology this means faster pathogen detection, broad-spectrum countermeasures, and better protective equipment before further transmissibility research. It is a coherent principle and faces an obvious coordination problem, since sequencing requires agreement among actors who do not agree. It is also the field's answer to the objection that its position amounts to a blanket [[precautionary-principle]]: the proposal is to change the order in which capabilities arrive, not to stop them arriving.The field's characteristic policy proposal is [[differential-technological-development]], which deliberately advances defensive and stabilising capabilities ahead of destabilising ones instead of accelerating or halting technology in general. Applied to biology this means faster pathogen detection, broad-spectrum countermeasures, and better protective equipment before further transmissibility research. The principle is coherent but faces an obvious coordination problem, since sequencing requires agreement among actors who do not agree. It is also the field's answer to the objection that its position amounts to a blanket [[precautionary-principle]]: the proposal changes the order in which capabilities arrive rather than stopping them from arriving. Existing instruments are thin. The Biological Weapons Convention has no verification mechanism, and nuclear arms control has eroded. AI governance assembled rapidly and lightly: the EU's AI Act entered into force in 2024, national AI safety and standards bodies were established in the UK and US from late 2023, and intergovernmental summits from Bletchley through Seoul and Paris produced declarations rather than obligations. The UN's 2024 Summit of the Future adopted a Pact for the Future including a declaration on future generations — the first UN document of that standing to carry an existential-risk framing, and non-binding.Existing instruments are thin. The Biological Weapons Convention has no verification mechanism, and nuclear arms control has eroded. AI governance has been assembled rapidly and lightly: the EU's AI Act entered into force in 2024, national AI safety and standards bodies were established in the UK and US from late 2023, and intergovernmental summits from Bletchley through Seoul and Paris produced declarations rather than obligations. The UN's 2024 Summit of the Future adopted a Pact for the Future that includes a declaration on future generations. It is the first UN document of that standing to carry an existential-risk framing, and it is non-binding. Planetary defence is the one area with a demonstrated capability: a spacecraft impact in 2022 measurably altered an asteroid's orbital period, establishing that deflection works for the class of object tested. It is also the least severe of the risks, which says something about the relation between tractability and importance here.Planetary defence is the one area with a demonstrated capability: a spacecraft impact in 2022 measurably altered an asteroid's orbital period, establishing that deflection works for the class of object tested. It is also the least severe of the risks, which says something about how tractability relates to importance here. ## Open problems## Open problems The unresolved question is whether existential risk can be made an ordinary policy category. Governments manage risks by frequency and expected cost, and a risk that has never occurred, cannot be insured, and would leave nobody to assign blame fits none of those procedures. The proposals for institutional fixes — standing bodies for future generations, mandatory long-horizon impact assessment, an international scientific panel on AI of the kind the UN General Assembly moved to establish in 2025 — are all attempts to create a constituency for outcomes that no living voter will experience. None has yet been shown to work.The unresolved question is whether existential risk can be made an ordinary policy category. Governments manage risks by frequency and expected cost, and a risk that has never occurred, cannot be insured, and would leave nobody to assign blame fits none of those procedures. The proposed institutional fixes include standing bodies for future generations, mandatory long-horizon impact assessment, and an international scientific panel on AI of the kind the UN General Assembly moved to establish in 2025. All are attempts to create a constituency for outcomes that no living voter will experience. None has yet been shown to work. ## See also## See also removed, struck through added, underlinedLine numbers count the serialised markdown of each revision, frontmatter included.
88 lines are the same in both revisions, and 37 of them are not shown. Neither revision here is necessarily the current article — read Existential risk for the text this wiki stands behind today.