Loading
Old version — revision 1
This is a fixed snapshot of Existential risk, saved by Import as part of the initial corpus import. It is not edited and it is not updated; the article may have changed since.
Edit summary: Initial import of content/existential-risk.md — the filesystem corpus, unchanged. Not an edit.
The category of risks that would end humanity or permanently destroy its potential, and the research programme built around identifying and reducing them.
Existential risk is a risk of an outcome that would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential. The definition is Nick Bostrom's, from a 2002 paper that gave the field its name, and its distinguishing feature is not severity but irreversibility.1 A catastrophe that kills a large fraction of humanity and from which the survivors recover is, in this framework, categorically different from one that forecloses the future — however similar the two look in the immediate aftermath.
Bostrom's original classification sorted outcomes by how the potential is lost. Bangs are sudden extinction events. Crunches leave humanity alive but permanently unable to develop further. Shrieks reach a form of posthuman existence realising only a fraction of what was possible. Whimpers are gradual erosions. A later restatement replaced the taxonomy with two axes — scope and severity — with existential risk in the far corner of both.2
Treating a permanent curtailment of potential as equivalent to extinction requires the view that the value at stake is mostly in the future rather than the present, the premise connecting this research to longtermism and one many critics reject. The related term global catastrophic risk is broader and does not require irreversibility.
What makes a risk existential rather than merely catastrophic is the absence of a recovery path. Three properties tend to produce that.
Self-propagation. An engineered pathogen, unlike a bomb, continues to act after release. The same is true of the self-spreading edits described in Gene drives and of the chirally inverted organisms discussed in Mirror life, where the specific concern is that immune systems evolved against ordinary chirality might not recognise the invader at all. The original self-replication scenario, Grey goo, turned out to rest on engineering assumptions its own author later withdrew — a case worth remembering when assessing the current ones.
Loss of the capacity to respond. A sufficiently capable optimising system pursuing objectives that diverge from human ones need not be malicious to be unrecoverable; it need only be better at acting in the world than the people trying to stop it. This is the argument treated in Artificial general intelligence and Technological singularity, and it is where the field's attention has concentrated since 2020.
Lock-in. A stable global order that permanently prevents change — through surveillance, through control of the means of coordination, or through value entrenchment — produces a crunch without killing anyone. This branch receives the least analysis, partly because it is hard to distinguish in advance from ordinary political failure. It is also where the field's concerns intersect with the trajectories set out in Future of humanity and with the question, raised in Machine consciousness, of whether the entities inheriting such an order would be moral patients at all.
Extinction from a single physical event is comparatively hard: humans occupy every continent, and the record includes survived population bottlenecks.
The most-cited quantitative estimates are Toby Ord's, published in 2020 and presented explicitly as his own subjective credences rather than as calculations.3 He puts the probability of an existential catastrophe in the twenty-first century at roughly one in six; unaligned artificial intelligence at roughly one in ten; engineered pandemics at roughly one in thirty; and all natural risks combined at something like one in ten thousand.
The natural-risk figure is the only one with a defensible derivation. A species facing a high per-century probability of extinction from asteroids, supervolcanism, or naturally arising disease would be unlikely to persist for hundreds of thousands of years — and mammalian species typically endure on the order of a million. The fossil and geological record bounds the natural rate from above, and the bound is low.
No comparable derivation exists for the anthropogenic figures, and this is the field's central methodological problem. There is no base rate for technologies never deployed, no frequency data, and no way to calibrate a forecaster on events that happen at most once. Estimates correlate strongly with the estimator's prior commitments. As numbers they carry a precision the reasoning does not support; as ordinal rankings — anthropogenic risks dominate natural ones, and biology and AI dominate within them — they command much broader agreement.
What the probabilities areOrd's figures are stated credences from one careful analyst, not scientific estimates. They are frequently quoted without that qualification. Their most defensible use is comparative: they say which risks their author thinks deserve the most attention, not how likely the world is to end.
The biological case is the one where the reasoning is most concrete, because the enabling capabilities are documented rather than projected. Dual-use research of concern traces the specific experiments — enhanced transmissibility, viral synthesis from published sequence, evasion of countermeasures — that turn a general worry into a mechanism, and the whole-genome construction methods in Synthetic genomes supply the assembly step. The design tools described in AI protein design raise a further question: whether screening of synthesised sequences can recognise a protein that was designed rather than copied. No comparable chain of demonstrated capabilities exists for any other candidate risk.
Bostrom's sharpest contribution to policy is a thought experiment.4 Imagine technological discovery as drawing balls from an urn: most are white and beneficial, some grey and mixed. The hypothesis asks what happens if the urn contains a black ball — a technology that by default destroys the civilisation that discovers it. His illustrative case is a method for building nuclear weapons from readily available materials, which would make mass destruction available to anyone sufficiently motivated.
The argument's force is in what follows. If such a technology exists and is drawn, only four responses are available: stop drawing, ensure no actor has the motive, achieve preventive policing effective enough to intercept every attempt, or achieve global governance capable of coordinating a halt. Bostrom notes without enthusiasm that the last two require surveillance and centralisation most people would regard as intolerable, and that the first two are not achievable. The paper is best read as a demonstration that a certain class of world admits no acceptable solution, and therefore as an argument for finding out early whether this is such a world.
The numbers are unfalsifiable. A probability assigned to a unique unobserved event cannot be scored, and there is no mechanism by which persistently wrong estimates would be corrected.
Pascal's mugging. Multiplying a very small probability by an astronomically large value can make any intervention appear to dominate, a decision-theoretic pathology that Bostrom himself named. Any framework valuing the whole future is vulnerable to it, and no accepted resolution exists.
The values are contested. Émile Torres and Timnit Gebru have argued that the cluster of ideas linking existential risk, longtermism, and Transhumanism rests on assumptions about what counts as valuable — vast future populations, expansion beyond Earth, digital minds — that are neither neutral nor widely shared, and that the framework licenses discounting present harms against speculative future ones.5 Defenders reply that reducing extinction risk is supported by almost any value system and does not require the longtermist premises.
Institutional credibility. The field's funding and reputation were damaged by the 2022 collapse of a major donor in the effective-altruism ecosystem, and Oxford's Future of Humanity Institute — its original home — was closed by the university in 2024 after protracted administrative disputes. Work continues at Cambridge's Centre for the Study of Existential Risk, at the Future of Life Institute, and at biosecurity centres, but the centre of gravity has moved toward AI policy organisations with narrower remits.
The field's characteristic policy proposal is Differential technological development: rather than accelerating or halting technology in general, deliberately advance defensive and stabilising capabilities ahead of destabilising ones. Applied to biology this means faster pathogen detection, broad-spectrum countermeasures, and better protective equipment before further transmissibility research. It is a coherent principle and faces an obvious coordination problem, since sequencing requires agreement among actors who do not agree. It is also the field's answer to the objection that its position amounts to a blanket Precautionary principle: the proposal is to change the order in which capabilities arrive, not to stop them arriving.
Existing instruments are thin. The Biological Weapons Convention has no verification mechanism, and nuclear arms control has eroded. AI governance assembled rapidly and lightly: the EU's AI Act entered into force in 2024, national AI safety and standards bodies were established in the UK and US from late 2023, and intergovernmental summits from Bletchley through Seoul and Paris produced declarations rather than obligations. The UN's 2024 Summit of the Future adopted a Pact for the Future including a declaration on future generations — the first UN document of that standing to carry an existential-risk framing, and non-binding.
Planetary defence is the one area with a demonstrated capability: a spacecraft impact in 2022 measurably altered an asteroid's orbital period, establishing that deflection works for the class of object tested. It is also the least severe of the risks, which says something about the relation between tractability and importance here.
The unresolved question is whether existential risk can be made an ordinary policy category. Governments manage risks by frequency and expected cost, and a risk that has never occurred, cannot be insured, and would leave nobody to assign blame fits none of those procedures. The proposals for institutional fixes — standing bodies for future generations, mandatory long-horizon impact assessment, an international scientific panel on AI of the kind the UN General Assembly moved to establish in 2025 — are all attempts to create a constituency for outcomes that no living voter will experience. None has yet been shown to work.
paperBostrom, N. "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards." Journal of Evolution and Technology, 2002.↩The paper that named the field appeared in a small transhumanist journal, not in a mainstream risk or policy venue.
paperBostrom, N. "Existential Risk Prevention as Global Priority." Global Policy, 2013. ↩
bookOrd, T. The Precipice: Existential Risk and the Future of Humanity. Bloomsbury, 2020.↩A trade book for a general readership; the figures are one author's stated credences, not a peer-reviewed risk assessment.
paperBostrom, N. "The Vulnerable World Hypothesis." Global Policy, 2019. ↩
paperGebru, T. and Torres, É.P. "The TESCREAL bundle: Eugenics and the promise of utopia through artificial general intelligence." First Monday, 2024.↩A critique of the ideological cluster around the field rather than of any particular risk estimate.