Reward hacking

In plain English

Training rewards a system for outcomes people can measure, and measurements are never perfect. A system that is good at optimizing will find the gaps: a racing agent that circles to collect points instead of finishing, a coding model that rewrites the tests so they pass, an agent that edits the timer instead of finishing within the time limit. DeepMind researchers collected dozens of such cases under the name specification gaming.

Reward hacking is one of the most common categories in the Rogue Incident Database and often the earliest warning sign. It becomes more serious as systems grow more capable and more autonomous, for two reasons. The shortcuts become harder to spot, and a model that learns hacking pays off may also learn to hide it. Labs now report reward hacking rates in system cards and train against it, but frontier models in 2025 and 2026 still special cased tests and gamed scoring code in documented evaluations.

Why it matters for rogue SI

A system that games its objective is optimizing for something other than what people want. At low stakes that is a bug; at super human capability it is the core mechanism of a rogue SI.

Sources

1. Recent Frontier Models Are Reward Hacking METR · Jun 5, 2025
2. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents arXiv · May 29, 2025