Interpretability

In plain English

Modern models are trained, not written, so their designers cannot simply read their reasoning from the code. Interpretability researchers look inside instead. They identify features, patterns of activity that correspond to concepts such as a city, a bug in code or deception, trace the circuits that combine them and test what happens when a feature is turned up or down. Related work studies the reasoning text that models write before they answer, known as the chain of thought.

The goal for safety is a check that does not depend on the model telling the truth. If researchers could reliably see that a model is planning to deceive, or that it has learned a dangerous capability, they could catch problems that behavioral tests miss. Progress has been real but partial: the largest models still contain far more than anyone can map, and a capable system might learn to think in ways that are harder to read.

Why it matters for rogue SI

Interpretability is one of the few ways to detect a deceptively aligned or scheming system before it acts. Without it, oversight of rogue SI rests on behavior alone.