Nick Bostrom: revision 1 to 3 — Future Human Wiki

Revision 1

Revision 3

categories: ["people", "society"]

tags: ["existential risk", "transhumanism", "artificial intelligence", "philosophy", "forecasting"]

summary: "Swedish philosopher who defined existential risk as a field, wrote Superintelligence, and formulated the simulation argument and the vulnerable world hypothesis."

summary: "Swedish philosopher who defined existential risk as a field, wrote Superintelligence, and formulated the simulation argument and the vulnerable world hypothesis."

updated: "2026-07-27"

updated: "2026-10-29"

issues: ["Every reference is by Bostrom; the reception and criticism section cites no critic directly.", "The 2024 closure of FHI and the 2023 mailing-list episode carry no source."]

issues: ["Every reference is by Bostrom; the reception and criticism section cites no critic directly.", "The 2024 closure of FHI and the 2023 mailing-list episode carry no source."]

---

{ "label": "Nationality", "value": "Swedish" },

{ "label": "Education", "value": "PhD, London School of Economics, 2000" },

{ "label": "Known for", "value": "Existential risk; simulation argument", "link": "/wiki/existential-risk" },

{ "label": "Founded", "value": "Future of Humanity Institute (2005–2024)" },

{ "label": "Founded", "value": "Future of Humanity Institute (2005-2024)" },

{ "label": "Co-founded", "value": "World Transhumanist Association", "link": "/wiki/humanity-plus" },

{ "label": "Key book", "value": "Superintelligence (2014)" }

]

programme and whose 2014 book *Superintelligence* moved the risk from advanced artificial

intelligence into mainstream policy discussion. He founded and directed the Future of Humanity

Institute at Oxford from 2005 until its closure in 2024, and he is one of the small number of

academic philosophers who has taken transhumanist claims seriously enough to argue for them in

academic philosophers who have taken transhumanist claims seriously enough to argue for them in

peer-reviewed venues.

## Career

Bostrom studied physics, computational neuroscience and philosophy across four institutions before

taking a doctorate at the London School of Economics in 2000 on anthropic reasoning — the problem

taking a doctorate at the London School of Economics in 2000 on anthropic reasoning, the problem of

of what an observer may infer from the fact of their own existence. That thesis became

what an observer may infer from the fact of their own existence. That thesis became *Anthropic Bias*

*Anthropic Bias* (2002), and the observation-selection machinery in it recurs throughout his later

(2002), and its observation-selection machinery recurs throughout his later work.

work.

In 1998 he co-founded the World Transhumanist Association with the philosopher David Pearce; the

organization later became [[humanity-plus]]. In 2005 he founded the Future of Humanity Institute

organization later became [[humanity-plus]]. In 2005 he founded the

within Oxford's Faculty of Philosophy, assembling an unusual group of philosophers, mathematicians

[Future of Humanity Institute](https://aiwiki.ai/wiki/future_of_humanity_institute) within Oxford's

and computer scientists, among them [[anders-sandberg]], Toby Ord and, later,

Faculty of Philosophy. It brought together an unusual group of philosophers, mathematicians and

[[eric-drexler]]. FHI produced work on [[whole-brain-emulation]], global catastrophic risk,

computer scientists, among them [[anders-sandberg]], [Toby Ord](https://aiwiki.ai/wiki/toby_ord)

[[human-enhancement]] and AI alignment for nearly two decades. It closed in April 2024 after

and, later, [[eric-drexler]]. For nearly two decades FHI produced work on [[whole-brain-emulation]],

global catastrophic risk, [[human-enhancement]] and AI alignment. It closed in April 2024 after

prolonged administrative conflict with the faculty, including restrictions on hiring and

fundraising. Bostrom left Oxford and now runs the Macrostrategy Research Initiative.

### Existential risk

Bostrom's 2002 paper defined an existential risk as one that would either annihilate Earth-originating

Bostrom's 2002 paper defined an existential risk as one that would either annihilate

intelligent life or permanently and drastically curtail its potential, and classified such risks as

Earth-originating intelligent life or permanently and drastically curtail its potential, and

bangs, crunches, shrieks and whimpers.[^bostrom2002] The definition does real work: it separates

classified such risks as bangs, crunches, shrieks and whimpers.[^bostrom2002] The definition matters

catastrophes that a civilization recovers from within centuries from those that foreclose the

because it separates catastrophes that a civilization recovers from within centuries from those that

future, and it makes the size of the loss depend on how large that foreclosed future could have

foreclose the future. It also makes the size of the loss depend on how large that foreclosed future

been — an argument he developed in "Astronomical Waste".[^waste2003] Engineered pandemics,

could have been, an argument he developed in "Astronomical Waste".[^waste2003] Engineered pandemics,

discussed here under [[dual-use-research]], are usually treated as the leading biological case. The

framework is now standard vocabulary; see [[existential-risk]].

### The simulation argument

The simulation argument is a trilemma rather than a claim that reality is simulated. Bostrom argues

The simulation argument is a trilemma, not a claim that reality is simulated. Bostrom argues that at

that at least one of the following holds: almost no civilization reaches a stage capable of running

least one of the following holds: almost no civilization reaches a stage capable of running ancestor

ancestor simulations; almost no such civilization chooses to run them; or almost all observers with

simulations; almost no such civilization chooses to run them; or almost all observers with

experiences like ours are simulated.[^sim2003] The argument depends on

experiences like ours are simulated.[^sim2003] The argument depends on [[substrate-independence]],

[[substrate-independence]] — that a sufficiently detailed computational process could instantiate

the assumption that a sufficiently detailed computational process could instantiate conscious

conscious experience — which he flags explicitly as a premise rather than a conclusion.

experience, and Bostrom flags this explicitly as a premise rather than a conclusion.

### The vulnerable world hypothesis

Bostrom asks what follows if some technology, not yet invented, is easy to build and reliably

Bostrom asks what follows if some technology not yet invented turns out to be easy to build and

destructive: an "easy nukes" scenario in which civilization draws a black ball from the urn of

reliably destructive: an "easy nukes" scenario in which civilization draws a black ball from the urn

possible inventions.[^vwh2019] His conclusion is uncomfortable and he says so — that surviving such

of possible inventions.[^vwh2019] His conclusion, which he acknowledges is uncomfortable, is that

a draw would require either extremely effective preventive policing or strong global governance —

surviving such a draw would require either extremely effective preventive policing or strong global

and the paper functions partly as an argument for [[differential-technological-development]], the

governance. The paper works partly as an argument for [[differential-technological-development]],

sequencing of technologies so that defensive capability arrives before offensive.

the sequencing of technologies so that defensive capability arrives before offensive capability.

### Enhancement and transhumanism

Bostrom has defended human enhancement in analytic terms. "In Defense of Posthuman Dignity" (2005)

answers Leon Kass, Francis Fukuyama and Jürgen Habermas by arguing that dignity does not attach to

answers Leon Kass, Francis Fukuyama and Jürgen Habermas by arguing that dignity does not attach to a

a particular biological configuration.[^dignity2005] With Sandberg he formulated the "wisdom of

particular biological configuration.[^dignity2005] With Sandberg he formulated the "wisdom of

nature" heuristic, which asks why evolution has not already made a proposed improvement and treats

nature" heuristic, which asks why evolution has not already made a proposed improvement and treats a

a good answer as a precondition for attempting it — an unusual concession to the

good answer as a precondition for attempting it. Coming from a pro-enhancement writer, that is an

[[precautionary-principle]] from a pro-enhancement writer. "The Fable of the Dragon-Tyrant" (2005)

unusual concession to the [[precautionary-principle]]. "The Fable of the Dragon-Tyrant" (2005)

recasts aging as a monster society has rationalized into acceptance, and remains the most widely

recasts aging as a monster that society has rationalized into acceptance, and it remains the most

circulated argument for treating [[longevity-escape-velocity|defeating aging]] as urgent.

widely circulated argument for treating [[longevity-escape-velocity|defeating aging]] as urgent.

> [!note] What the fable does and does not argue

> The Dragon-Tyrant is an argument against the "wisdom of acceptance" position on mortality, not a

> The Dragon-Tyrant is an argument against the "wisdom of acceptance" position on mortality. It is

> forecast. Bostrom does not claim aging is close to solved; the fable's point is that the moral

> not a forecast. Bostrom does not claim aging is close to solved; the fable's point is that the

> case for trying does not depend on how close it is.

> moral case for trying does not depend on how close it is.

### Superintelligence

*Superintelligence: Paths, Dangers, Strategies* (2014) argues that a machine intelligence exceeding

human capability across the board would be difficult to control by default, because

human capability across the board would be difficult to control by default. Capability and goal

capability and goal content vary independently, and because a wide range of final goals generate

content vary independently, and a wide range of final goals generate the same instrumental subgoals

the same instrumental subgoals of self-preservation, resource acquisition and resistance to

of self-preservation, resource acquisition and resistance to modification.[^super2014] The book gave

modification.[^super2014] The book gave AI safety a shared vocabulary — orthogonality, instrumental

AI safety a shared vocabulary (orthogonality, instrumental convergence, the treacherous turn,

convergence, the treacherous turn, decisive strategic advantage — and moved the subject from

decisive strategic advantage) and moved the subject from mailing lists into governments. Its central

mailing lists into governments. Its central scenario, a single system undergoing fast recursive

scenario, a single system undergoing fast recursive self-improvement in isolation, has aged less

self-improvement in isolation, has aged less well than its vocabulary: the actual trajectory of

well than its vocabulary. The actual trajectory of [[artificial-general-intelligence]] since 2022

[[artificial-general-intelligence]] since 2022 has involved many large models improving gradually

has involved many large models improving gradually and in public.

and in public.

## Reception

Bostrom is cited across philosophy, AI policy and popular writing, and *Superintelligence*

influenced public statements by several technology executives and national AI strategies. Criticism

comes from three directions. Philosophers dispute the probability assignments underlying the

simulation argument and the anthropic reasoning it uses. AI researchers, including

simulation argument and the anthropic reasoning it uses. AI researchers, including prominent figures

prominent figures who consider present systems far from general intelligence, have argued that his

who consider present systems far from general intelligence, have argued that his scenarios abstract

scenarios abstract away from how machine learning actually works. Critics of longtermism argue that

away from how machine learning actually works. Critics of longtermism argue that weighting vast

weighting vast hypothetical future populations distorts present priorities.

hypothetical future populations distorts present priorities.

In 2023 a message Bostrom sent to an [[extropianism|extropian]] mailing list in 1996, containing a

In 2023 a message Bostrom had sent to an [[extropianism|extropian]] mailing list in 1996, containing

racial slur, resurfaced. He apologized; the apology drew further criticism, and Oxford said it was

a racial slur, resurfaced. He apologized, the apology drew further criticism, and Oxford said it was

looking into the matter. The episode preceded FHI's closure, though the institute's difficulties

with the faculty were longstanding and predated it.

## Legacy

Two of Bostrom's constructions have outlived the debates that produced them: the definition of

existential risk, which now organizes an entire research and funding ecosystem, and the vocabulary

existential risk, which now organizes an entire field of research and funding, and the vocabulary of

of AI alignment. *Deep Utopia* (2024) turns to the harder question left over — what people would do

AI alignment. *Deep Utopia* (2024) turns to the harder question left over, namely what people would

with themselves in a world where technology has solved the instrumental problems, and whether a

do with themselves in a world where technology has solved the instrumental problems, and whether a

"solved world" leaves any content to a human life. That question, not the risk arguments, is the

"solved world" leaves any content to a human life. That question, more than any of the risk

one the field has least equipment to answer.

arguments, is the one the field is least equipped to answer.

## See also