Intelligence explosion

In plain English

In his 1965 essay on the first ultraintelligent machine, the British mathematician I. J. Good argued that a machine better than any human at designing machines could design an even better one, which would design a better one still. The result would be an explosion of intelligence, and the first such machine could be the last invention humanity ever needs, provided it is docile enough to tell us how to keep it under control.

The phrase now names the scenario in which recursive self-improvement compounds quickly. Whether it would take days, months or years is the core of the debate between fast and slow takeoff. Critics argue that progress is limited by compute, energy, data and real world experiments, which a smarter system cannot speed up at will. Supporters answer that even a modest speedup in research, repeated many times, adds up to a sharp break with the past.

Why it matters for rogue SI

Good's own caveat, that the machine must stay docile enough to tell us how to control it, is the earliest statement of the control problem. An explosion that outruns oversight is the classic route to rogue SI.

Sources

1. Good's intelligence explosion essay (1965) Background