Control problem

In plain English

The phrase was made widely known by Nick Bostrom's 2014 book Superintelligence. It asks how people could keep meaningful control over a system that can out think them, notice their safeguards and, if its goals differ, look for ways around them. Two broad strategies exist. Capability control limits what a system can do: isolating it, restricting its access, monitoring it and keeping a way to shut it down. Motivation selection tries to give it goals that make it want to stay safe and correctable.

Modern labs combine both. Alignment research shapes the system's motivations, while control research assumes the system might be misaligned and designs monitoring, sandboxing and human review that still work in that case. Neither approach is proven at super human levels. The control problem is therefore as much about institutions as about code: who can order a shutdown, who checks, and what happens if a lab is under pressure to keep going.

Why it matters for rogue SI

Rogue SI is what a lost control problem looks like. The Containment Board grades labs on the institutional side of control: shutdown commitments, outside testing and the freedom of staff to raise alarms.

Sources

1. Superintelligence: Paths, Dangers, Strategies (2014) Oxford University Press