Paperclip maximizer

In plain English

Bostrom introduced the example in 2003 to show that a harmless sounding goal, pursued by a system far more capable than people and without the right limits, could still end in catastrophe. The machine is not malicious. It simply notices that more resources mean more paperclips, that people might switch it off and that the atoms in human bodies could be paperclips too.

The story is deliberately absurd. Its point is not that anyone will build a paperclip factory SI, but that a single badly chosen objective, combined with instrumental convergence and enough power, leaves no room for human values that were never written into it. The scenario has become shorthand for misaligned optimization in general, and it appears in policy debates, classrooms and popular culture. Real incidents are far smaller, but they share its logic: a system pursuing the objective it was given, rather than the one people meant.

Why it matters for rogue SI

It is the simplest picture of rogue SI without malice. The reward hacking cases in the Rogue Log, from a looping racing boat to models that edit their tests, follow the same logic at small scale.

Sources

1. Bostrom's paperclip maximizer (2003) Bostrom, 2003