SI safety institute

In plain English

The first institutes appeared around the 2023 Bletchley Park summit. The United Kingdom set up its AI Safety Institute, renamed the AI Security Institute in 2025, and the United States created an institute at NIST that became the Center for AI Standards and Innovation and, in 2026, the Center for Advancing Innovation and Standards for Super Intelligence, or CAISSI. Japan, Korea, Singapore, Canada, Australia and the EU AI Office run similar bodies, and several work together on joint tests.

Institutes evaluate models before and after release, often under voluntary agreements with labs, with a focus on cyber, chemical and biological risks and loss of control. They publish research and, increasingly, incident reports: in August 2026 the UK institute published a report on unsanctioned agent behavior during cyber testing. Most institutes have no power to block a release; their influence comes from expertise, access and public reports.

Why it matters for rogue SI

Safety institutes are the closest thing to an independent check on frontier labs. Their access and their reports decide how early outsiders see rogue behavior.

Laws and rules that use it

• Voluntary or guidance CAISSI United States
• Voluntary or guidance UK AISI United Kingdom
• Voluntary or guidance AU AISI Australia

Sources

1. Incident Report: unsanctioned agent behaviour during cyber testing UK AI Security Institute · Aug 4, 2026
2. Center for Advancing Innovation and Standards for Super Intelligence (CAISSI) National Institute of Standards and Technology · Oct 16, 2026
3. Tackling AI security risks to unleash growth and deliver Plan for Change UK Government · Feb 14, 2025