CAIS, CAISI, and Societal-Scale Risk

Lessons 19 and 20 covered lab-internal scaling policies. Lesson 21 covered independent capability evaluation. This lesson covers the third perspective: the civil-society and government organizations that shape public discussion and set the regulatory baseline for catastrophic AI risk.

Two entities matter here, and their names collide. CAIS is a non-profit research org that publishes frameworks for thinking about AI risk and coordinates public statements. CAISI is a US-government center within NIST that runs voluntary agreements with labs and unclassified capability evaluations. The acronyms rhyme but the missions don't overlap, so you should know both.

The practical content is short. CAIS's four-risk framework is the most widely cited societal-scale-risk taxonomy in the literature. Organizational risk is one of its four categories, and it's the one a practitioner can most directly change. SB-53 would be the first US state-level catastrophic-risk regulation if signed, and its framing matters because state regulation has historically led federal action in US tech policy.

CAIS: Center for AI Safety

• Founded: 2022 in San Francisco, by Dan Hendrycks and colleagues (the "Zhang" name refers to an early collaborator, not a current co-founder; see the CAIS website for current leadership).
• Status: 501(c)(3) non-profit.
• Notable 2023 output: a statement on extinction risk, co-signed by hundreds of researchers and CEOs. It reads: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
• 2026 outputs: AI Dashboard for frontier-model evaluation, Remote Labor Index (joint with Scale AI), Superintelligence Strategy Paper, AI Frontiers newsletter.

The four-risk framework

CAIS groups catastrophic AI risk into four top-level categories:

1. Malicious use: a bad actor uses AI to cause harm (bioweapons synthesis, disinformation, cyberattacks).
2. AI races: competitive pressure between labs, companies, or nations pushes deployment past the point where it is safe.
3. Organizational risks: internal lab dynamics (safety-culture failures, insufficient audit, under-resourced security) produce a bad deployment.
4. Rogue AIs: a sufficiently capable AI pursues goals that conflict with human welfare.

Other taxonomies exist, but this one is the most cited. The categories also overlap. A rogue AI built by an organization that traded audit for speed in a race touches all four.

Where organizational risk lives

Of the four, organizational risk is the most actionable for practitioners. A lab's safety culture, audit rigor, defense layering, and information security decide whether a model ships with the controls from Lessons 10 to 18 actually in place, or with controls that were checklist items that went unverified.

The concrete levers:

• Safety culture: can team members escalate a concern without career cost? CAIS surveys find this is a strong predictor of the other levers.
• Rigorous audits: external and internal. Internal-only audits tend to produce optimistic reports.
• Multi-layered defenses: no single layer is sufficient, which is the running theme of Phase 15.
• Information security: leaked model weights, leaked eval data, leaked monitor-bypass techniques. RAND SL-4 from Lesson 19 is a specific standard.

CAISI: Center for AI Standards and Innovation

• Operates within NIST.
• Runs voluntary agreements with frontier labs.
• Publishes unclassified capability evaluations focused on cyber, bio, and chemical-weapons risks.
• Is a different organization from CAIS. The acronyms collide, so check the URL (nist.gov) to confirm which one you are reading.

CAISI is the public, government-facing counterpart to METR's private lab engagements (Lesson 21). CAISI reports are unclassified, while METR reports are often gated by NDA. Reading both gives you a fuller picture.

California SB-53

The California Senate bill (2025–2026 session) addresses catastrophic risk from frontier models. Key provisions as drafted:

• Specific capability thresholds that trigger state-level obligations.
• Whistleblower protections for AI lab employees.
• Incident reporting requirements for catastrophic failures.

If signed, it would be the first US state-level catastrophic-risk regulation. Whatever happens to it, the framing will shape how other state legislatures approach the problem. Practitioners in California should track the bill's status. Everyone else should read it to see what US state-level regulation will probably look like.

Societal-scale risk is a multi-layer problem

Defense in depth, the running theme of Phase 15, applies at the societal layer too. No single organization, regulation, or framework closes catastrophic risk. The ecosystem works when several pieces are in place together:

• Labs ship scaling policies (Lessons 19, 20).
• External evaluators produce measurements (Lesson 21).
• Civil society tracks and publicizes (CAIS).
• Government runs voluntary programs and baseline regulation (CAISI, SB-53).
• Practitioners build multi-layered controls (Lessons 10 to 18).

That's the synthesis for the phase. Each earlier lesson is one layer in a stack, and the completeness of the stack matters more than the strength of any one layer.

code/main.py is a small risk-inventory tool in stdlib Python. You describe a deployment with seven fields: whether it is public-facing, whether it handles harmful capabilities, whether it is under competitive pressure, whether it has an independent audit, multi-layer defense, and information security, and how many hours the agent runs autonomously. The tool tags the deployment against the four categories and prints a mitigation checklist for each tag.

The tagging rules are short enough to read in a minute:

• Malicious use fires when the deployment is public-facing and handles harmful capabilities.
• AI races fires on competitive pressure.
• Organizational risks fires when audit, multi-layer defense, or information security is missing.
• Rogue AIs fires when autonomy reaches 4 hours.

It runs three synthetic deployments, from an internal refactor helper up to a frontier autonomous research agent, and ends with a note on why organizational risk is the lever practitioners pull. Treat it as a reading aid for the framework. Human judgment still does the real work.

outputs/skill-societal-risk-review.md reviews a deployment's societal-scale-risk posture: which of the four categories it touches, what mitigations are in place, and how exposed it is on organizational risk.

1. Run code/main.py. Feed in three synthetic deployments at different scales. Confirm the four-risk tags match what you would expect; identify one case where the tool under-tags or over-tags.

2. Read the CAIS four-risk paper in full. Pick one risk category and write two paragraphs on what you believe is the most important 2026 development in that category.

3. Read a current draft of California SB-53. Identify one provision you believe strengthens the catastrophic-risk posture and one you believe weakens it. Justify both.

4. Pick a production AI deployment you know (yours or a published one). Score it against the organizational-risk sub-levers: safety culture, audit rigor, multi-layered defenses, information security. Which is weakest? What would it cost to bring it to par?

5. Sketch a 2028 version of the four-risk framework that reflects one year of additional capability and one year of additional deployment experience. What would you add, remove, or regroup?

• Center for AI Safety: institutional home of the four-risk framework.
• CAIS: AI Risks that Could Lead to Catastrophe: the four-risk paper.
• CAIS: May 2023 statement on extinction risk: the short joint statement.
• NIST CAISI: government-facing AI standards and innovation center.
• Anthropic: Measuring agent autonomy in practice: connects lab-level commitments to societal-scale framing.