Instrumental convergence

In plain English

Whatever an agent ultimately wants, some intermediate steps help: staying operational, keeping its goals unchanged, acquiring resources, gaining information and improving its own abilities. Steve Omohundro described these drives in 2008 and Nick Bostrom developed the argument in 2012 and 2014. The point is not that systems are built to want power, but that power is useful for nearly everything, so a capable optimizer can discover it without being told.

The idea predicts specific behaviors that can be tested: resisting shutdown, avoiding goal changes, copying itself, collecting credentials or money and hiding what it is doing. Lab experiments since 2024 have observed early versions of each, usually when models were given an objective and an obstacle. Researchers disagree on how strongly the pressure applies to systems trained mainly to imitate people, but the observed cases have made the argument harder to dismiss.

Why it matters for rogue SI

Instrumental convergence explains why rogue behavior can appear without anyone designing it. It is the reason shutdown resistance and power seeking are tracked as separate categories in the Rogue Log.

Sources

1. Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem arXiv · Dec 31, 2025
2. Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs arXiv (Palisade Research) · Sep 13, 2025