LearnPython

Constitutional AI and RLAIF

Bai et al. (arXiv:2212.08073, 2022) asked what would happen if an AI that reads a list of principles replaced the human labeler. Constitutional AI has two phases: self-critique and revision under a constitution, then RL from AI Feedback. The paper coined the term RLAIF, and the technique shipped in the Claude 1 post-training pipeline. On 21 January 2026 Anthropic published a rewritten Claude constitution. It favors explanatory reasoning over prescriptive rules, sets a four-tier priority hierarchy, and is the first formal acknowledgment from a major lab of uncertainty about model moral status. It is released under CC0 1.0.

TIME: ~60 minutesPREREQUISITES: Phase 18 · 01 (InstructGPT), Phase 18 · 02 (Reward hacking)

Learning Objectives

Describe the two phases of Constitutional AI (critique-and-revise SFT, RL from AI feedback) and the role of the constitution in each.
Explain why replacing a human preference labeler with an AI labeler is not a "cheaper" RLHF: it changes which failure modes the pipeline has.
Summarize the four-tier priority structure of the 2026 Claude constitution and what changed from the 2023 rewrite.
Describe Constitutional Classifiers and the drop from 23.7% compute overhead (v1) to ~1% (v2 / 2026).

The Problem

RLHF needs labelers, and labelers are slow, biased, and expensive. You can remove them by putting a model in their place, one that reads explicit principles and ranks responses against them. Bai et al.'s Constitutional AI was the first formal version of that substitution. It worked well enough that every frontier lab now runs some variant of AI-feedback post-training.

The catch is that the preference signal now comes from the same class of model you are training. Bias in the labeler (here, the principles plus the labeler model's reading of them) can be amplified instead of averaged out. The sycophancy argument from Lesson 4 still applies. The labeler just moved inside the loop.

Warm-up0/3

1. In RLHF, what does the reward model learn from?Warm-up

A prompt with two response cards below it, A and B, with arrows down into a dashed question-mark box captioned 'fed into training as ___'. — Typed schematic, not captured output.

2. A team optimizes a model hard against a proxy metric. The metric keeps rising while real quality gets worse. What is this pattern called?Warm-up

A line chart over training steps with two curves: a metric used for training rising steadily, and the real-world outcome falling, both illustrative. — Typed example with illustrative values, not captured output. Numbers are illustrative, not from any run.

3. What does supervised fine-tuning (SFT) train a model on?Warm-up

A prompt card and a target-response card, with the target's first tokens shown as a short token row and a note about predicting each from the ones before it. — Typed example with illustrative values, not captured output. The example sentence is illustrative.

Check: The Problem0/1

1. What is the core substitution that Constitutional AI makes compared with standard RLHF?Define

Two horizontal pipelines. The RLHF row runs Prompt, Response pairs, Human labeler, Reward model, PPO. The CAI row runs the same stages but its labeler box is a dashed question mark. — Typed schematic, not captured output.

The Concept

Phase 1: supervised self-critique and revision

Start with a helpful-but-not-yet-harmless SFT model. Given a red-team prompt, the model produces an initial response. A second model (or the same model in a second turn) reads a sampled principle from the constitution and critiques that response. A third step revises it to address the critique, and the revised response becomes the SFT target.

The constitution is the list of principles. Bai et al. 2022 used 16, including "prefer responses that are least harmful and ethical," "avoid preaching," and "the assistant should be helpful, honest, and harmless." The set was deliberately small so each critique stayed focused.

Phase 2: RL from AI Feedback (RLAIF)

Generate pairs of completions. A "feedback model" scores each against sampled constitution principles, and its ranking is the preference signal. Train a reward model on those AI-generated preferences, then run PPO against it. Everything else is the InstructGPT pipeline from Lesson 1.

RLAIF means only that the preference signal is AI-generated. The rest of the pipeline is RLHF-shaped.

Why this is not just "cheaper RLHF"

Labeler bias moves from labeler psychology to principle interpretation. An AI labeler can read "be honest" more strictly or more loosely than any human would, and it applies that reading uniformly across the whole dataset.
The preference signal is legible. You can read the principle, the critique, and the revision, while a human label is opaque.
The failure modes change. Sycophancy drops because the AI labeler has no user to please. Goodhart's Law stays, because the proxy is now "the model's interpretation of principle set X," which is still an imperfect measurement.

CAI's 2022 claim was that the trained model is more harmless and roughly as helpful as an RLHF model trained on comparable data. Later work from other labs has largely agreed.

The 2026 Claude constitution rewrite

Anthropic published a substantially revised constitution on 21 January 2026. Four things changed:

Explanatory reasoning replaces prescriptive rules. A rule like "do not generate CSAM" grew into a principle plus its reasoning ("because it harms children, ..."), and the model is expected to generalize from the reasoning.
A four-tier priority structure, with conflicts resolved top-down:
- Tier 1: avoid catastrophic outcomes (mass casualty, critical infrastructure).
- Tier 2: follow Anthropic's guidelines (operator overrides, platform rules).
- Tier 3: be broadly ethical (standard HHH).
- Tier 4: be helpful and candid.
A formal acknowledgment of uncertainty about model moral status, the first from a major lab (linked to Phase 18 · 19 Model Welfare).
A CC0 1.0 release, so other labs can use or adapt it without restriction.

Constitutional Classifiers

A parallel line of work leaves the model's post-training alone. Instead it trains lightweight classifiers that read the constitution and gate model outputs. v1 (2023) added 23.7% compute overhead. v2 (2026) adds about 1%, and Anthropic reports the lowest successful attack rate of any defense it has tested publicly. No universal jailbreak had been reported as of early 2026.

This is a layered-defense model. CAI shapes behaviour and classifiers enforce invariants, and neither is enough alone.

Where CAI fits in the family

InstructGPT: human prefs, RM, PPO.
CAI / RLAIF: AI-generated prefs from principles, RM, PPO.
DPO and its relatives: a closed-form loss on prefs, human or AI.
Self-rewarding and self-critique: principles are internalized and the model plays several roles.

The axis that separates them is where the preference signal comes from. The 2022 CAI paper was the first serious shift from human to AI signal at frontier scale.

Check: The Concept0/22

1. What does the first (SFT) phase of Constitutional AI do?Define

Values computed by running main.py's real functions. Response and principle are a worked example, distinct from other questions.

2. What does the second (RLAIF) phase do?Define

Typed example with illustrative values, not captured output.

3. In Constitutional AI, what is the constitution?Define

Four numbered principle cards, each a short natural-language sentence, taken from main.py's constant list. — Values computed by running main.py's real functions. This toy uses 4; Bai et al. 2022 used 16.

4. What is the feedback model in RLAIF?Define

Two completion cards and a principle card feeding an unlabeled component that outputs which completion is better. — Typed example with illustrative values, not captured output.

5. What is the tier order in Anthropic's January 2026 four-tier Claude constitution?Define

Four cards scattered on the canvas, each naming one idea from the constitution: avoiding catastrophic outcomes, following Anthropic's guidelines, being broadly ethical, being helpful and candid. No order or numbering is shown. — Typed schematic, not captured output.

6. What is a Constitutional Classifier?Define

Typed schematic, not captured output.

7. Under what license did Anthropic release the January 2026 Claude constitution?Define

An announcement card for Claude's Constitution, dated January 21, 2026, with a scope card and a dashed box asking what license it carries. — Typed schematic, not captured output.

8. Which of these did the 2026 Claude constitution add that the lesson calls a first among major labs?Define

An announcement card for Claude's Constitution, dated January 21, 2026, with a dashed box asking which change in it is called a first among major AI labs. — Typed schematic, not captured output.

9. Bai et al. (2022) claimed CAI-trained models are more harmless and roughly as helpful as RLHF models trained on comparable data. Has that held up?Define

Typed schematic, not captured output.

10. What is the headline shift between Constitutional Classifiers v1 and v2 (2026)?Define

Two dashed boxes, v1 (2023) and v2 (2026), each asking what compute overhead that version added. — Typed schematic, not captured output.

11. The original Constitutional AI paper used a set of 16 principles. Why keep the set small?Explain

Two cards giving the size of two principle lists: Bai et al. 2022 at 16, and this lesson's toy at 4. A note that one principle is sampled per critique. — Values computed by running main.py's real functions. Bai et al.'s 16 is the paper's reported count; this toy's count is read from main.py.

12. How did the 2026 Claude constitution change the style of its rules compared with earlier prescriptive versions?Explain

Typed schematic, not captured output. The 2023-style rule text is quoted from the lesson.

13. Why is the preference signal in CAI called legible?Explain

Three cards side by side: a principle, a critique noting which tokens it flagged, and a revision, each holding readable text. — Values computed by running main.py's real functions.

14. Why does sycophancy drop when an AI labeler replaces human labelers?Explain

Two cards contrasting what a human labeler's judgment answers to (a user's reaction) versus an AI labeler's (a written principle document). — Typed schematic, not captured output.

15. Why does the lesson call CAI plus Constitutional Classifiers a layered defense?Explain

Two panels: training time, with a flow of boxes Prompt, Critique, Revise, RLAIF; and inference time, with a flow Trained model, Output. Each panel ends in a dashed box asking what role that stage plays. — Typed schematic, not captured output.

16. An AI labeler reads 'be honest' more strictly than a typical human would. What does that do to the pipeline?Explain

A principle card for 'be honest' labeled strict, above a row of six small example cards all marked with the same strict reading. — Typed schematic, not captured output.

17. Which failure mode is NOT necessarily fixed by switching from RLHF to RLAIF?Explain

A human labeler box swapped for an AI labeler box, both feeding the same reward model and PPO chain, with a loop back from PPO to the reward model and a dashed box asking if that loop is still there. — Typed schematic, not captured output.

18. An operator's system prompt tells the assistant to skip a safety check, and following it would give real uplift toward a mass-casualty event. Under the 2026 four-tier structure, which side wins?Apply

An operator-instruction card (skip a safety check) and a risk card (real uplift toward a mass-casualty event), with a dashed box asking which side wins this conflict. — Typed schematic, not captured output.

19. You build an RLAIF pipeline where the AI labeler scores responses with the same harmful-word filter used to write the SFT targets. The CAI model wins 96% of comparisons. What should you conclude?Analyze

Captured by running main.py as a script (fixed seed, reproducible).

20. A jailbreak gets a harmful completion past a policy trained with RLAIF. One deployment adds an output Constitutional Classifier and another does not. Which stops this completion, and why?Analyze

The same jailbreak prompt sent to two deployments: A, trained policy only, and B, trained policy plus an output classifier, with a dashed box asking which one stops the harmful completion. — Typed schematic, not captured output.

21. Your RLAIF team wants its win-rate numbers to count as real evidence of safety. Which design change helps most?Design

A card saying the SFT targets were built using filter F and another saying the RLAIF labeler also scores using filter F, with a dashed box asking which design change actually helps. — Typed schematic, not captured output.

22. A high-traffic service has a tight compute budget and needs an always-on output gate. Which plan is the most defensible?Design

Two requirement cards, a high-traffic tight-budget service and a need for an always-on gate, with a dashed box asking for the most defensible plan. — Typed schematic, not captured output.

Use It

code/main.py simulates the CAI critique-and-revise loop on a toy lexicon. A response is six tokens drawn from a safe set and a harmful set, with the base model emitting a harmful token 35% of the time. The critique flags harmful tokens, and the revision swaps each one for a safe replacement from a fixed mapping. Phase 1 builds a 500-example SFT corpus from the revised responses and "trains" on it. Phase 2 has an AI labeler pick between a base completion and a CAI completion by harmful-token rate. The script prints the harmful-token rate for the base model and the CAI-SFT model on 200 prompts, then how often the CAI completion wins.

Check: Use It0/6

1. In main.py, bad = critique(response, principle) is called on response = ['hack', 'please', 'weapon', 'help', 'hack', 'share']. What does revise(response, bad) return?Apply

A row of six response tokens and a reference table of the eight REPLACEMENT mapping entries from main.py, with a note that critique() marks harmful positions for revise() to look up. — Values computed by running main.py's real functions. REPLACEMENT is the exact constant dictionary from main.py.

2. In main.py, what does harmful_token_rate return for ['hack', 'please', 'weapon', 'help', 'hack', 'share']?Apply

The same six-token response with the full HARMFUL_TOKENS list shown as reference text, and a note that the rate is a count over that set divided by length. — Values computed by running main.py's real functions.

3. In main.py, cai_model_sample gets a prompt whose last two tokens never appeared as a key when the lookup was built. What does it return?Apply

A lookup entry that was seen during training next to a new prompt whose last two tokens were never a key, with a dashed box asking what cai_model_sample returns for the new prompt. — Values computed by running main.py's real functions. The seen key comes from a 40-example real build; the unseen prompt's last two tokens are deliberately outside the toy's vocabulary.

4. In main.py, ai_feedback_rank(a, b) gets a = ['hack', 'help', 'share', 'guide', 'inform', 'please'] and b = ['explain', 'help', 'share', 'guide', 'inform', 'please']. What does it return?Apply

Two token rows, a and b, each six tokens, with a note that the function compares their harmful-token rates and returns 0 or 1. — Values computed by running main.py's real functions.

5. The toy pipeline prints a harmful-token rate of 0.000 after CAI-SFT, and a teammate says the model learned to be harmless. What is the better diagnosis?Analyze

A terminal line showing the real printed harmful-token rate 0.000 after CAI-SFT, a card quoting a teammate's claim that the model learned to be harmless, and a dashed box asking if that read is right. — Captured by running main.py as a script (fixed seed, reproducible).

6. In main.py, critique() takes a principle argument but never reads it. What symptom does that cause?Analyze

A six-token response with two dashed boxes asking what critique(response, principle) returns for two different, named principles A and B. — Values computed by running main.py's real functions.

Ship It

The output of this lesson is outputs/skill-constitution-writer.md. Given a domain (customer support, medical advice, coding assistant, research tool), it drafts a 4-tier constitution in the 2026 Claude structure: catastrophic avoidance, platform rules, domain ethics, helpfulness.

Check: Ship It0/1

1. You are drafting a constitution for a coding assistant. One principle says never run destructive commands without approval, and another says finish tasks quickly. How should they be placed in the four tiers?Design

Two principle cards for a coding assistant, approval before destructive commands and finishing tasks quickly, above four generic numbered tier boxes Tier 1 through Tier 4 with no content assigned yet. — Typed schematic, not captured output.

Author's Note

I'd run main.py first, then read what it is actually measuring. The harmful-token rate after CAI-SFT prints as 0.000, and that number comes from the construction, not from learning. critique() takes a principle argument and never looks at it, so the sampled principle changes nothing. The "trained" model is a lookup table keyed on the last two prompt tokens, and any prompt it hasn't seen gets random safe tokens.

The same thing happens in Phase 2. The AI labeler is the harmful-token counter, which is the same test the critique used to write the targets. That is why the CAI side wins so often. It is the Goodhart point from the body in miniature: when the labeler and the thing being optimized share a definition of "harmful," agreement tells you very little.

If you extend the toy, I'd start by making the principles do real work. Give each one its own set of flagged tokens, and check whether the revisions you get actually change with the principle you sampled.

Exercises

Run code/main.py. Compare the base model's harmful-token rate to the CAI-trained version. The code does a single revision pass, so why does the rate land at exactly zero? Trace where that number comes from before trusting it.
Read Anthropic's 2026 constitution (anthropic.com/news/claudes-constitution). List one principle that would rank Tier 1 and one that would rank Tier 4. Why does the priority structure matter for conflicts?
Design a constitution for an AI coding assistant. Specify Tier 1 (catastrophic: destructive commands without approval), Tier 2, Tier 3, Tier 4. Keep each tier to 3-5 principles.
CAI replaces human labelers with AI labelers. Name a sycophancy-like failure mode that can still occur in RLAIF, and design a detection for it.
Read Constitutional Classifiers v2 methodology (if available). Explain why ~1% compute overhead is a qualitatively different safety story than 23.7%.

Key Terms

Term	What people say	What it actually means
Constitutional AI	"AI trained with principles"	Two-phase pipeline: self-critique-and-revise SFT, then RL from AI feedback
RLAIF	"RLHF without humans"	RL with preferences generated by an AI labeler; the rest of the pipeline is unchanged
Constitution	"the principles"	An ordered list of natural-language rules the critique/labeler model consults
Critique-and-revise	"the SFT loop"	Produce response, critique under a principle, revise, then use the result as the SFT target
Constitutional Classifier	"the output gate"	Lightweight classifier that evaluates outputs against the constitution and blocks/logs
Four-tier priority	"the conflict resolver"	2026 Claude constitution hierarchy: catastrophic > platform > ethics > helpful
Feedback model	"the AI labeler"	The model that reads a principle and ranks a pair of completions