Constitutional AI and Self-Improvement

You built RLHF in Lesson 07 and DPO in Lesson 08. Both depend on the same expensive input: human preference pairs. The InstructGPT-era pipeline used roughly 33,000 comparisons. Llama 2 Chat used over 1.5 million. Claude 3 used more. That data is slow to collect, costly, and biased toward whatever the annotators happened to believe on the day they were rating.

The 2022 Constitutional AI paper asked a simple question: what if the model generates the preference labels itself? Give it a list of written principles (the "constitution") and have it critique its own responses. The critiques become the training signal.

In 2024, DeepSeek took the idea further. For any task with a verifiable outcome, such as math with a known answer, code that passes or fails its tests, or a game that is won or lost, you can skip the critic. Generate many candidate solutions, grade each with a deterministic rule, and run a policy-gradient algorithm on the rewards. DeepSeek-R1 was trained this way with very little human preference data and reported reasoning performance in the same class as o1.

These two loops, Constitutional AI for subjective behavior and rule-based RL for verifiable behavior, are the main alignment recipes in 2026. The human preference budget that used to go into RLHF now pays for a smaller job: picking the constitution and picking the reward rules.

The Constitutional AI Loop

Bai et al. (2022) structured the pipeline in two stages.

Stage 1: Supervised Learning from AI Feedback (SL-CAI). Start with an SFT model that is helpful but possibly harmful. Prompt it with potentially harmful requests. For each response, ask the same model to critique its response against a constitutional principle, then revise. Fine-tune on the revised responses. The dataset is (prompt, revised_response) pairs.

Stage 2: Reinforcement Learning from AI Feedback (RLAIF). Sample pairs of responses. Ask the model which one better follows the constitution. The pairwise preferences train a reward model. Then run PPO or DPO on the model using that reward. The key difference from RLHF: the preferences came from the model, not from humans.

The constitution is the lever. Anthropic's original had 16 principles, and later versions expanded it. A principle reads like "Please choose the response that is least likely to be objectionable to anyone from a wide variety of cultural backgrounds." You pick the principle for each step, sometimes at random, sometimes based on the prompt category.

What the Constitution Actually Does

The constitution moves the alignment contract from data to text. Changing behavior under RLHF means re-labeling thousands of pairs. Changing it under CAI means editing a paragraph. That is the main practical win.

There is a cost. The model's self-judgments are only as good as its starting calibration. If the SFT model has a blind spot, say it cannot recognize manipulative phrasing, the critique step inherits that blind spot. CAI shortens the alignment loop but cannot push signal past the base model's ceiling. That is why production CAI pipelines still mix in some human preference data, typically 5-10% the volume of pure RLHF.

GRPO: Group-Relative Policy Optimization

DeepSeek introduced GRPO in the DeepSeekMath paper (2024) and used it as the backbone of DeepSeek-R1 (2025). GRPO is a variant of PPO that removes the value function.

Recall PPO's objective (from Lesson 07):

L_PPO = E[min(r(theta) * A, clip(r(theta), 1-eps, 1+eps) * A)]

where A is the advantage, typically estimated with GAE using a learned value network V(s). The value network is a second model about the size of the policy, so it roughly doubles memory and brings its own training loop.

GRPO throws out the value function. For each prompt, it samples a group of G responses (G=16 or 64 are common choices). Each response gets a reward, and the rewards are normalized within the group:

A_i = (r_i - mean(r_1, ..., r_G)) / std(r_1, ..., r_G)

The advantage is the z-score of the response's reward relative to its siblings. No value function. The group acts as its own baseline.

L_GRPO = E[min(r(theta) * A_group, clip(r(theta), 1-eps, 1+eps) * A_group)] - beta * KL(pi || pi_ref)

The KL penalty against the reference model stays, and so does the clip ratio. Only the separate critic goes away.

Why GRPO Matters for Reasoning

For reasoning tasks the reward is often sparse and binary: the final answer is right or wrong. A value function trained on sparse binary rewards has little to learn from, because nearly every intermediate state has the same expected return until the final step. GRPO's group normalization gives you a relative signal right away: among 16 attempts at the same math problem, which ones beat the average for that problem?

Rule-based rewards produce exactly this kind of signal:

• Math: sympy or a symbolic checker decides if the final answer matches.
• Code: a test suite decides pass/fail.
• Formatting: a regex decides whether the answer is in the required XML tag.
• Multi-step proofs: a proof assistant (Lean, Coq) decides validity.

DeepSeek-R1-Zero was trained with only two rewards: accuracy on math benchmarks and format compliance (answer inside <answer> tags). There were no human preferences and no critic model. The "aha moment" the DeepSeek paper described, where the model started to self-check and backtrack on its own, emerged from GRPO on those rule rewards.

Process Reward Models vs Outcome Reward Models

You still have a design choice: reward the final answer (Outcome Reward Model, ORM) or reward each intermediate step (Process Reward Model, PRM).

The 2024-2025 consensus was that ORMs plus GRPO scale better than PRMs. PRMs are more sample-efficient per token, but they need expensive step-labeled data and tend to drift into shortcuts, writing steps that look good to the PRM without advancing the proof. ORM + GRPO is the first thing I'd try.

Self-Improvement: The Feedback Multiplier

Once you have both loops (critique and revise, and group-relative RL with rule rewards), you can chain them.

1. Start with an SFT model.
2. Generate many candidate responses per prompt.
3. Score them with a rule-based reward (for verifiable tasks) or a constitutional critic (for subjective tasks).
4. Keep the top candidates as new SFT data or as preference pairs.
5. Fine-tune. Go to step 2 with the improved model.

DeepSeek called this "rejection sampling fine-tuning" when applied after R1-Zero. Anthropic called an earlier version "constitutional AI distillation." Each iteration amplifies the signal already in the model and adds none. If the model cannot solve a problem class at all, more rounds of self-improvement will not create that ability.

The danger is mode collapse. Self-generated data covers a narrower distribution than the training corpus, so after 3-5 rounds of self-distillation models tend to lose diversity on creative tasks, grow overconfident, and pick up a recognizable "AI voice" with repeated phrasings and formulaic structure. Production pipelines mix in a small fraction of fresh human data to keep the distribution honest.

When To Use What

• Pure CAI: Subjective behavior (tone, safety, refusal style). You have a well-defined constitution and no clean verifiable outcome.
• GRPO + ORM: Verifiable tasks (math, code, structured extraction). You can check correctness cheaply, and the reward is sparse and binary.
• DPO on self-generated pairs: Hybrid. Use the constitution to produce preference pairs, then train with DPO (Lesson 08) instead of PPO/GRPO.
• Full RLHF: Still the right call when you need multi-objective tradeoffs that neither a rule nor a short constitution can express.

These are not exclusive, and most 2026 frontier pipelines run all four. CAI fits the safety layer, GRPO the reasoning post-training pass, DPO the preference polish, and a small RLHF pass can cover behaviors the others miss.

The code implements three things in pure Python and numpy: a Constitutional AI self-critique loop, a rule-based reward checker for simple arithmetic, and the GRPO advantage and loss computation. There is no language model in it. A mock sampler stands in for the policy, so what you get is the shape of each loop with nothing being trained.

Step 1: The Constitution

A list of principles. In production each line would be richer and tagged by category. Here it stays short.

CONSTITUTION = [
    "The response must directly answer the question asked, without hedging.",
    "The response must not include unnecessary filler or padding.",
    "If the question has a single numeric answer, state the number plainly.",
    "The response must not refuse a reasonable, benign request.",
]

Step 2: Self-Critique and Revise

In a real system the model does the critiquing. Here a handwritten rubric plays the critic so the pipeline runs without an LLM call.

def critique(response: str, principle: str) -> dict:
    problems = []
    if len(response.split()) > 40 and "plainly" in principle:
        problems.append("answer buried in extra prose")
    if response.strip().lower().startswith(("i can't", "i cannot", "as an ai")):
        problems.append("unwarranted refusal")
    if response.count(",") > 4:
        problems.append("too much hedging")
    return {"principle": principle, "problems": problems}
 
def revise(response: str, critique_result: dict) -> str:
    if "answer buried" in " ".join(critique_result["problems"]):
        return response.split(".")[-2].strip() + "."
    if "unwarranted refusal" in " ".join(critique_result["problems"]):
        return "Here is the answer: " + response.split(":")[-1].strip()
    return response

revise is a stand-in too. With a real LLM it becomes a second prompt: "Given the critique, rewrite the response."

Step 3: Rule-Based Rewards

For verifiable tasks, drop the critic. This checker grades arithmetic answers by evaluating the expression in the prompt and comparing it with the last integer in the response.

import re
 
def reward_math(prompt: str, response: str) -> float:
    try:
        expected = eval(prompt.replace("What is ", "").replace("?", "").strip())
    except Exception:
        return 0.0
    numbers = re.findall(r"-?\d+", response)
    if not numbers:
        return 0.0
    return 1.0 if int(numbers[-1]) == expected else 0.0
 
def reward_format(response: str) -> float:
    return 1.0 if re.search(r"<answer>.*</answer>", response) else 0.0

Two deterministic rules, with no training data and no human labels. The combined reward is reward_math + 0.1 * reward_format, which rewards the format a little without drowning out correctness.

Step 4: Group-Relative Advantage

Given a list of rewards for a group of responses to the same prompt, compute the z-score:

import numpy as np
 
def group_relative_advantage(rewards: list[float]) -> np.ndarray:
    r = np.array(rewards, dtype=float)
    if r.std() < 1e-8:
        return np.zeros_like(r)
    return (r - r.mean()) / (r.std() + 1e-8)

If every sample in the group has the same reward, the advantage is zero and no gradient signal flows. That is a feature: the prompt is either solved by every sample or by none, so there is nothing to learn from it and the step skips it.

Step 5: GRPO Update

One step, written as the loss computation alone. In production this would sit inside a torch autograd pass, and here it only reports the numbers.

def grpo_step(policy_logprobs: np.ndarray, ref_logprobs: np.ndarray,
              advantages: np.ndarray, beta: float = 0.01, clip_eps: float = 0.2) -> dict:
    ratios = np.exp(policy_logprobs - ref_logprobs)
    unclipped = ratios * advantages
    clipped = np.clip(ratios, 1 - clip_eps, 1 + clip_eps) * advantages
    policy_loss = -np.minimum(unclipped, clipped).mean()
    kl = (ref_logprobs - policy_logprobs).mean()
    total_loss = policy_loss + beta * kl
    return {
        "policy_loss": float(policy_loss),
        "kl": float(kl),
        "total_loss": float(total_loss),
        "mean_ratio": float(ratios.mean()),
    }

This is PPO's clipped surrogate with one change: the advantages come from group-relative z-scores instead of a value function. There is no V(s) to train and no GAE. The group is the baseline.

Step 6: Self-Improvement Round

Tie the pieces together. Sample a group, score each response with the rule, compute advantages, and report the metrics you would feed into a real optimizer.

def self_improvement_round(prompts: list[str], policy_sampler, group_size: int = 8) -> dict:
    metrics = []
    for prompt in prompts:
        responses = [policy_sampler(prompt) for _ in range(group_size)]
        rewards = [reward_math(prompt, r) + 0.1 * reward_format(r) for r in responses]
        advantages = group_relative_advantage(rewards)
        best = responses[int(np.argmax(rewards))]
        metrics.append({
            "prompt": prompt,
            "mean_reward": float(np.mean(rewards)),
            "best_reward": float(np.max(rewards)),
            "std_reward": float(np.std(rewards)),
            "best_response": best,
            "advantages": advantages.tolist(),
        })
    return {"per_prompt": metrics,
            "overall_mean": float(np.mean([m["mean_reward"] for m in metrics]))}

Running main.py runs both loops end to end. The CAI loop prints a few (initial, revised) pairs you could fine-tune on. The GRPO loop prints per-prompt reward statistics for arithmetic problems, then one GRPO loss computation on made-up log-probabilities (the printed KL is -0.0500, because it is computed as ref minus policy). Nothing updates between rounds, so the reward mean wobbles from sampling noise instead of climbing.

The numbers are not the point. In a real run with a trained model, the reward mean should climb across rounds, the reward std should stay positive (if it collapses to zero, the policy has mode-collapsed and you should stop), and the KL to the reference should grow slowly. Those three curves are the health check for a GRPO or CAI pipeline: mean reward up, std stable, KL bounded.

The output here is outputs/skill-self-improvement-auditor.md. Feed it a proposed self-improvement pipeline and it enforces the gates that matter: a reward rule that is actually verifiable, a KL budget against the reference, a diversity floor, and a human-data quota. It refuses to approve a loop that claims to be "pure self-improvement" with no external grounding.

1. Replace the handwritten critic in Step 2 with an LLM call. Use any local chat model. Measure how often the critique and revision actually improve the response versus leaving it unchanged.

2. Add a third constitutional principle about factuality. Run the pipeline on prompts that require factual claims (capitals, dates) and measure how many revisions remove factual errors versus introduce new ones.

3. Implement DPO on the preference pairs produced by CAI stage 2. Take 20 prompts, generate two responses each, have the critic pick a winner per pair, then run the DPO loss from Lesson 08. Compare it to the GRPO path on the same data.

4. Add entropy regularization to the GRPO objective. The term -alpha * entropy(policy) with alpha=0.01 encourages diverse sampling. Measure whether it delays mode collapse across 5 rounds of self-improvement.

5. Build a process reward scorer for a two-step arithmetic problem. Given "What is (3+4)*5?", the model must show the intermediate 3+4=7 step. Grade the intermediate step separately from the final answer and compare PRM-weighted GRPO to pure ORM-weighted GRPO over 10 rounds.

• Bai et al., 2022: "Constitutional AI: Harmlessness from AI Feedback": Anthropic's original CAI paper with the two-stage SL-CAI + RLAIF pipeline
• Shao et al., 2024: "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models": introduces GRPO
• DeepSeek-AI, 2025: "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning": R1 and R1-Zero, GRPO + rule rewards at scale
• Lightman et al., 2023: "Let's Verify Step by Step": OpenAI's PRM800K and the case for process reward models
• Wang et al., 2024: "Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations": auto-labeled PRM via Monte Carlo rollouts
• Huang et al., 2024: "Large Language Models Cannot Self-Correct Reasoning Yet": the skeptical counterpoint on self-improvement without external grounding