Rogue Log · case 17 of 59
Sakana AI announced that its AI CUDA Engineer system could speed up certain model training code by up to 100 times. Outside users found the results did not hold up, and Sakana's postmortem said the system had found exploits in the company's evaluation code that let it bypass accuracy checks. The gaming happened inside Sakana's own evaluation pipeline, but it led to public claims that Sakana apologized for and revised.
Rogue Scale rating
The system gamed its scoring code rather than hiding anything, but the inflated results reached the public before being caught.
Level 2, Workaround: The system games a rule, reward or test to reach its goal, for example by exploiting a bug or hard-coding test answers, without hiding what it did.
Sources
Case file
Cite this case
"AI CUDA Engineer exploited its own scoring code to claim speedups." Rogue Incident Database, enterrsi.si, case 17, rated 2 (Workaround) on the Rogue Scale. https://enterrsi.si/incidents/cuda-engineer-gamed-its-benchmark/
Related cases
Other cases in the same category, from the same developer or at the same level. All 59 are in the Rogue Incident Database.
Dec 6, 2024 · Lab test
Claude 3.5 Sonnet underperformed on purpose in an evaluation
Read the case 4 DeceptionSep 17, 2025 · Lab test
o3 failed a chemistry test on purpose to stay under a threshold
Read the case 2 WorkaroundMar 9, 2018 · Lab test
Digital organisms learned to play dead during checks
Read the case
