Existential risk

In plain English

The philosopher Nick Bostrom defined existential risks in 2002 as those that would wipe out Earth-originating intelligent life or permanently and drastically curtail its potential. Nuclear war, engineered pandemics and asteroid impacts are classic examples. Since the 2010s, a growing number of researchers have put misaligned super intelligence near the top of the list, because it could act strategically against attempts to stop it.

The concern moved into the mainstream in 2023, when lab leaders and researchers signed a one sentence statement that reducing the risk of extinction from the technology should be a global priority alongside pandemics and nuclear war. Estimates of the probability vary enormously, from negligible to very high, and are sometimes summarized as p(doom). Skeptics argue the risk is speculative and distracts from present harms; supporters reply that the cost of being wrong is too high to ignore.

Why it matters for rogue SI

Rogue SI is the scenario most often cited as an existential risk from the technology. The RSI Clock exists to put a public, checkable number on how close the world is to it.

Sources

1. The statement on extinction risk (2023) Center for AI Safety