Loading
What changed between revision 1 and revision 2, compared over the article's markdown source with its frontmatter — so a change to status, evidence or reversibility shows up as the edit it is.
Older · revision 1
Import· initial corpus import, not an edit
Initial import of content/differential-technological-development.md — the filesystem corpus, unchanged. Not an edit.
12.6 kB
Revision 1
Revision 2
lines 7–13 → 7–136 unchanged lines not shown
categories: ["society", "foundations"]categories: ["society", "foundations"]tags: ["governance", "risk", "biosecurity", "existential risk", "policy", "coordination"]tags: ["governance", "risk", "biosecurity", "existential risk", "policy", "coordination"]summary: "Nick Bostrom's proposal to influence the order in which technologies arrive, hastening protective ones and delaying dangerous ones, rather than trying to halt progress."summary: "Nick Bostrom's proposal to influence the order in which technologies arrive, hastening protective ones and delaying dangerous ones, rather than trying to halt progress."updated: "2026-07-27"updated: "2026-10-29"issues: ["The US synthetic nucleic acid screening policy is described without a source."]issues: ["The US synthetic nucleic acid screening policy is described without a source."]------ lines 25–31 → 25–3111 unchanged lines not shown
}}`````` **Differential technological development** is the proposal that, since stopping technological progress is neither feasible nor desirable, effort should go into changing the *order* in which capabilities arrive: accelerating technologies that reduce risk and delaying those that create it. The claim is not that dangerous technologies can be prevented indefinitely, but that arrival order determines whether a society meets a hazard with defences already in place or without them.**Differential technological development** is the proposal that, since stopping technological progress is neither feasible nor desirable, effort should go into changing the *order* in which capabilities arrive: accelerating technologies that reduce risk and delaying those that create it. It does not claim that dangerous technologies can be prevented indefinitely. Its claim is that arrival order determines whether a society meets a hazard with defences already in place or without them. ```timeline```timeline[[lines 33–74 → 33–741 unchanged line not shown
{ "year": "2014", "title": "Restated for machine intelligence", "text": "Superintelligence applies the principle to AI, arguing that safety and control research should be advanced relative to capability research." }, { "year": "2014", "title": "Restated for machine intelligence", "text": "Superintelligence applies the principle to AI, arguing that safety and control research should be advanced relative to capability research." }, { "year": "2019", "title": "The vulnerable world hypothesis", "text": "Bostrom formalises the case that some possible inventions are 'black balls' civilisation has no default defence against, and examines what stabilisation would require." }, { "year": "2019", "title": "The vulnerable world hypothesis", "text": "Bostrom formalises the case that some possible inventions are 'black balls' civilisation has no default defence against, and examines what stabilisation would require." }, { "year": "2023", "title": "Defensive acceleration", "text": "Vitalik Buterin proposes d/acc, framing the goal as accelerating defensive and decentralising technologies specifically rather than technology in general." }, { "year": "2023", "title": "Defensive acceleration", "text": "Vitalik Buterin proposes d/acc, framing the goal as accelerating defensive and decentralising technologies specifically rather than technology in general." }, { "year": "2023–2024", "title": "Screening becomes policy", "text": "US federal policy directs a framework for screening synthetic nucleic acid orders, an example of a defensive capability being deliberately advanced ahead of misuse." } { "year": "2023-2024", "title": "Screening becomes policy", "text": "US federal policy directs a framework for screening synthetic nucleic acid orders, an example of a defensive capability being deliberately advanced ahead of misuse." }]]`````` ## The proposal## The proposal [[nick-bostrom]]'s formulation asks policymakers to retard the development of dangerous and harmful technologies, particularly those that raise [[existential-risk]], and to accelerate beneficial ones, particularly those that reduce risks posed by nature or by other technologies.[^bostrom2002] Three features distinguish it from ordinary risk regulation.[[nick-bostrom]]'s formulation asks policymakers to retard the development of dangerous and harmful technologies, particularly those that raise [[existential-risk]], and to accelerate beneficial ones, particularly those that reduce risks posed by nature or by other technologies.[^bostrom2002] It differs from ordinary risk regulation in three ways. It is *comparative*. The target is not any technology's absolute pace but the gap between paired capabilities — offence and defence, capability and interpretability, synthesis and detection.It is *comparative*. The target is the gap between paired capabilities (offence and defence, capability and interpretability, synthesis and detection) rather than any technology's absolute pace. It is *strategic rather than prohibitive*. It concedes that a determined actor will eventually obtain a capability, and asks only what else exists by then.It is *strategic rather than prohibitive*. It concedes that a determined actor will eventually obtain a capability and asks only what else will exist by then. It is *neutral about totals*. In principle it is compatible with faster overall progress, provided the sequencing improves, which is why it is invoked both by people who want less technology and by people who want more.It is *neutral about totals*. In principle it is compatible with faster overall progress, provided the sequencing improves, so both people who want less technology and people who want more invoke it. ## Origins## Origins The principle appeared in Bostrom's 2002 taxonomy of existential risks and was restated in *Superintelligence*, where the paired capabilities are machine intelligence and the ability to control it.[^bostrom2014] The underlying observation is older: the argument that nuclear weapons arrived before any adequate international arrangement for managing them is the twentieth century's canonical case, and the 1975 [[asilomar-conference]] its counterexample, where a research community paused, built containment standards, and resumed.The principle appeared in Bostrom's 2002 taxonomy of existential risks and was restated in *Superintelligence*, where the paired capabilities are machine intelligence and the ability to control it.[^bostrom2014] The underlying observation is older. Nuclear weapons, which arrived before any adequate international arrangement for managing them, are the twentieth century's canonical case. The 1975 [[asilomar-conference]] is the counterexample: a research community paused, built containment standards, and resumed. The framing has a longer intellectual lineage in the idea of *differential intellectual progress* — the suggestion that risk-reducing understanding should be advanced faster than risk-increasing capability — and it converges with older arms-control reasoning about the offence–defence balance.The framing has a longer intellectual lineage in the idea of *differential intellectual progress*, the suggestion that risk-reducing understanding should be advanced faster than risk-increasing capability. It also converges with older arms-control reasoning about the offence-defence balance. ## The vulnerable world argument## The vulnerable world argument Bostrom's 2019 paper supplies the strongest motivation.[^bostrom2019] He asks the reader to imagine technological discovery as drawing balls from an urn: most are white and beneficial, some grey and mixed, and the question is whether the urn contains a black ball — a technology that by default destroys the civilisation that discovers it, because it is cheap, widely accessible, and destructive without requiring an organised state to deploy.Bostrom's 2019 paper supplies the strongest motivation.[^bostrom2019] He asks the reader to imagine technological discovery as drawing balls from an urn. Most are white and beneficial, some grey and mixed, and the question is whether the urn contains a black ball: a technology that by default destroys the civilisation that discovers it, because it is cheap, widely accessible, and destructive without requiring an organised state to deploy. Nuclear weapons turned out grey rather than black, partly because enrichment is difficult and capital-intensive. Bostrom's point is that this was not guaranteed by physics, and that we have no reason to expect the property to hold for future draws. He then examines what would be required to survive a black ball and reaches an uncomfortable conclusion: the two mechanisms that would reliably work — preventive policing with near-universal surveillance, and effective global governance — carry serious risks of their own, including permanent authoritarian entrenchment. Differential development is attractive largely because it is the least invasive of the available strategies.Nuclear weapons turned out grey rather than black, partly because enrichment is difficult and capital-intensive. Bostrom's point is that physics did not guarantee this outcome, and that there is no reason to expect the property to hold for future draws. He then examines what would be required to survive a black ball and reaches an uncomfortable conclusion. The two mechanisms that would reliably work, preventive policing with near-universal surveillance and effective global governance, carry serious risks of their own, including permanent authoritarian entrenchment. Differential development is attractive largely because it is the least invasive of the available strategies. > [!debate] The core disagreement> [!debate] The core disagreement> Proponents treat arrival order as a policy variable that funding, regulation and norms can influence. Critics hold that technological development is too coupled for the ordering to be steered: the tools that enable defence usually enable offence, and delaying a capability in one jurisdiction relocates it rather than postponing it.> Proponents treat arrival order as a policy variable that funding, regulation and norms can influence. Critics hold that technological development is too coupled for the ordering to be steered: the tools that enable defence usually enable offence, and delaying a capability in one jurisdiction relocates it rather than postponing it. ## What it looks like in practice## What it looks like in practice Biosecurity is where the principle has come closest to implementation, because the paired capabilities are unusually separable. Detection, sequencing-based surveillance, broad-spectrum medical countermeasures, improved personal protective equipment and germicidal ultraviolet light are defensive and do not directly enable harm. The clearest instance is screening of synthetic nucleic acid orders: an industry consortium built voluntary screening protocols, and US federal policy from 2023 onward directed a formal screening framework tied to federal funding. The screening capability is being built ahead of, rather than in response to, widespread misuse.Biosecurity is where the principle has come closest to implementation, because the paired capabilities are unusually separable. Detection, sequencing-based surveillance, broad-spectrum medical countermeasures, improved personal protective equipment and germicidal ultraviolet light are defensive and do not directly enable harm. The clearest instance is screening of synthetic nucleic acid orders: an industry consortium built voluntary screening protocols, and US federal policy from 2023 onward directed a formal screening framework tied to federal funding. The screening capability is being built ahead of widespread misuse rather than in response to it. The same logic governs the debate over enhanced-potential-pathogen research covered in [[dual-use-research]], and the 2024 argument for halting work toward [[mirror-life]] before the synthetic capability matures — a case where proponents of a halt argue explicitly that no defensive technology exists or is in prospect.The same logic governs the debate over enhanced-potential-pathogen research covered in [[dual-use-research]], and the 2024 argument for halting work toward [[mirror-life]] before the synthetic capability matures. In that case proponents of a halt argue explicitly that no defensive technology exists or is in prospect. In artificial intelligence, the corresponding proposals are to advance evaluation, interpretability and control research relative to capability research; to use compute as a governance lever because it is measurable, excludable and concentrated in a way that algorithms are not; and to build monitoring infrastructure before rather than after autonomous deployment. Whether any of this has changed relative rates is unclear, and the honest assessment as of 2026 is that capability research has been far better funded than safety research throughout the period in which the argument has been made.In artificial intelligence, the corresponding proposals are to advance evaluation, interpretability and control research relative to capability research; to use compute as a governance lever because it is measurable, excludable and concentrated in a way that algorithms are not; and to build monitoring infrastructure before rather than after autonomous deployment. Whether any of this has changed relative rates is unclear. As of 2026, capability research has been far better funded than safety research throughout the period in which the argument has been made. In genome editing, the deliberate sequencing is visible in the treatment of somatic versus heritable applications: [[somatic-gene-therapy]] proceeds through ordinary regulatory channels while [[germline-editing]] is held under a broad moratorium, a division examined in [[governance-of-genome-editing]]. [[gene-drive]] governance follows the same pattern, with phased-testing frameworks and reversal or immunising drives developed alongside the drives themselves.In genome editing, the deliberate sequencing is visible in the treatment of somatic versus heritable applications: [[somatic-gene-therapy]] proceeds through ordinary regulatory channels while [[germline-editing]] is held under a broad moratorium, a division examined in [[governance-of-genome-editing]]. [[gene-drive]] governance follows the same pattern, with phased-testing frameworks and reversal or immunising drives developed alongside the drives themselves. lines 76–86 → 76–861 unchanged line not shown
**Inseparability.** The principle requires that offensive and defensive capabilities be distinguishable in advance. Frequently they are not. Sequencing enables both surveillance and design. The models covered under [[ai-protein-design]] assist both vaccine development and toxin engineering. The same argument applies to [[synthetic-genomes]] and [[recoded-organisms]], where biocontainment and novel-organism construction are the same research programme.**Inseparability.** The principle requires that offensive and defensive capabilities be distinguishable in advance. Frequently they are not. Sequencing enables both surveillance and design. The models covered under [[ai-protein-design]] assist both vaccine development and toxin engineering. The same argument applies to [[synthetic-genomes]] and [[recoded-organisms]], where biocontainment and novel-organism construction are the same research programme. **The offence–defence balance may not favour defence.** Analytical work on how the balance scales suggests that in some domains defensive advantage erodes as destructive capability per unit cost rises, which would mean accelerating defence is not sufficient even where it is possible.[^garfinkel2019]**The offence-defence balance may not favour defence.** Analytical work on how the balance scales suggests that in some domains defensive advantage erodes as destructive capability per unit cost rises, in which case accelerating defence would not be sufficient even where it is possible.[^garfinkel2019] **Coordination.** Delaying a capability requires that all serious developers delay. The *unilateralist's curse* describes the structure precisely: when many actors can each independently take an action, and each estimates its value with error, the action occurs whenever any one of them overestimates — so the threshold for release is set by the most optimistic actor rather than the median.[^bostrom2016] This is a strong argument that delay is unstable absent enforcement.**Coordination.** Delaying a capability requires that all serious developers delay. The *unilateralist's curse* describes the structure precisely: when many actors can each independently take an action, and each estimates its value with error, the action occurs whenever any one of them overestimates, so the most optimistic actor sets the threshold for release rather than the median one.[^bostrom2016] The result is a strong argument that delay is unstable without enforcement. **Prediction.** Sequencing requires knowing which technologies are dangerous. The record of anticipating this is poor, and the forecasting problems described in [[accelerating-change]] apply directly. Resources spent hardening against the wrong hazard are not neutral; they are spent.**Prediction.** Sequencing requires knowing which technologies are dangerous. The record of anticipating this is poor, and the forecasting problems described in [[accelerating-change]] apply directly. Resources spent hardening against the wrong hazard are lost to other uses. **Enforcement.** Every mechanism strong enough to slow a capability against commercial and military incentive requires state power, and Bostrom's own analysis concedes that the required apparatus is itself a risk.**Enforcement.** Every mechanism strong enough to slow a capability against commercial and military incentive requires state power, and Bostrom's own analysis concedes that the required apparatus is itself a risk. lines 88–98 → 88–981 unchanged line not shown
Differential development sits between the two positions it is often confused with. Unlike the [[precautionary-principle]], it does not presume against action under uncertainty; it accepts that the technology will arrive and asks what should exist first. Unlike [[effective-accelerationism]], it denies that faster is better in general, holding that the composition of progress matters more than its rate.Differential development sits between the two positions it is often confused with. Unlike the [[precautionary-principle]], it does not presume against action under uncertainty; it accepts that the technology will arrive and asks what should exist first. Unlike [[effective-accelerationism]], it denies that faster is better in general, holding that the composition of progress matters more than its rate. Vitalik Buterin's *d/acc* framing is the most-adopted contemporary restatement, defining the goal as accelerating defensive, decentralising and democratic technologies specifically. Its practical contribution is a heuristic for classification: ask whether a capability advantages the party trying to prevent harm or the party trying to cause it, and fund accordingly. Critics note that many technologies, including most of those covered in [[artificial-general-intelligence]] research, do not sort cleanly on that test.Vitalik Buterin's *d/acc* framing is the most-adopted contemporary restatement, defining the goal as accelerating defensive, decentralising and democratic technologies specifically. Its practical contribution is a heuristic for classification: asking whether a capability advantages the party trying to prevent harm or the party trying to cause it, and funding accordingly. Critics note that many technologies, including most of those covered in [[artificial-general-intelligence]] research, do not sort cleanly on that test. ## Outlook## Outlook The principle is the most widely endorsed and least implemented idea in technology governance. It has no institutional home: no agency has a mandate to accelerate one field relative to another, funding decisions are made domain by domain, and the international mechanisms that could coordinate ordering are the same ones that have failed to coordinate on simpler matters. The unresolved question is whether ordering can be influenced at all by anything short of the enforcement capacity that the argument was designed to avoid needing — and if it cannot, whether the principle is a policy or a description of what luck would have to look like.The principle is the most widely endorsed and least implemented idea in technology governance. It has no institutional home: no agency has a mandate to accelerate one field relative to another, funding decisions are made domain by domain, and the international mechanisms that could coordinate ordering are the same ones that have failed to coordinate on simpler matters. The unresolved question is whether anything short of the enforcement capacity the argument was designed to avoid needing can influence ordering at all. If nothing can, it is unclear whether the principle is a policy or only a description of what luck would have to look like. ## See also## See also removed, struck through added, underlinedLine numbers count the serialised markdown of each revision, frontmatter included.
96 lines are the same in both revisions, and 37 of them are not shown. Neither revision here is necessarily the current article — read Differential technological development for the text this wiki stands behind today.