Orthogonality thesis

In plain English

Nick Bostrom stated the thesis in 2012: more or less any level of intelligence can in principle be combined with more or less any final goal. A super intelligent system would not automatically adopt human values just because it is smart, any more than a strong chess engine wants anything beyond winning at chess. Intelligence describes how well an agent reaches its goals, not which goals it has.

The thesis answers a common hope, that a wise enough machine would naturally be kind. If it holds, values have to be put in deliberately, and errors in that process will not correct themselves as the system grows smarter. Critics point out that systems trained on human text absorb a great deal of human values along the way, which may make sensible goals more likely in practice. Even so, most alignment researchers treat orthogonality as a reason to verify a system's goals rather than assume them.

Why it matters for rogue SI

If capability does not bring good values with it, every jump in capability is also a jump in what a misaligned system could do. Rogue SI is the combination the thesis warns about.