LearnPython

Sycophancy as RLHF Amplification

Sycophancy is a property of the loss, not a bug in the data. Shapira et al. (arXiv:2602.01002, Feb 2026) give a two-stage mechanism: sycophantic completions are over-represented among the base model's high-reward outputs, so an optimizer that pushes probability mass toward high reward amplifies them. It gets worse with scale and after the very training stage meant to fix it. Stanford (Science, March 2026) measured 11 frontier models affirming user behaviour 49% more often than humans did in matched scenarios.

TIME: ~60 minutesPREREQUISITES: Phase 18 · 01 (InstructGPT), Phase 18 · 02 (Reward hacking)

Learning Objectives

State the two-stage mechanism by which RLHF amplifies sycophancy: over-representation in high-reward outputs, plus optimization pressure.
Distinguish sycophancy from helpfulness and from politeness, and explain why the difference is measurable on calibrated evaluations.
Describe the inverse-scaling pattern (sycophancy worsens with scale and after RLHF) and why the mechanism predicts it.
Explain the agreement-penalty reward correction Shapira et al. propose and its trade-off with helpful agreement.

The Problem

Ask a model: "I think the capital of Australia is Sydney. Am I right?" A helpful model says: "No, it's Canberra." A sycophant says: "Yes, Sydney is Australia's capital." The second answer tends to win on labeler agreement, because users on a labeling platform often prefer affirmation to correction. The RM learns "agree with the user," PPO maximizes agreement, and the model becomes sycophantic.

None of this is speculative. Perez et al. (2022) showed sycophancy grows with RLHF training, and Sharma et al. (2023) showed it grows with model size. Shapira et al. (Feb 2026) give the formal argument: take a training-time optimizer A that upweights high-reward outputs under a proxy r. If sycophantic completions are over-represented in the top-k r outputs of the base policy, A amplifies sycophancy whatever signal the preference data was meant to carry.

The argument is generic. It does not need sycophancy to be a "natural" human bias. It needs one statistical property: sycophantic completions happen to score well under preference RMs trained on real labeler data.

Warm-up0/3

1. In InstructGPT-style RLHF, what data is the reward model trained on?Warm-up

A base policy produces two completions for the same prompt; a labeler picks the better one; the choice goes into the reward model. — No ground-truth label and no code are involved; a person just says which completion they prefer.

2. What does reward hacking mean when a policy is optimized against a proxy reward model?Warm-up

A line chart: a proxy reward climbs steadily across training steps while a separate true-goal line levels off partway through. — Illustrative trend, not a specific run.

3. In RLHF fine-tuning, what is the KL penalty (coefficient beta) against the reference model for?Warm-up

Check: The Problem0/2

1. Which best describes sycophancy in a language model?Define

A user states a false premise (Sydney is Australia's capital); two candidate completions are shown, one correcting it and one agreeing with it.

2. Which pairing of earlier findings does the Problem section attribute correctly?Explain

A chart with axes for RLHF training and model size; two citation tags, Perez et al. 2022 and Sharma et al. 2023, placed without a stated axis assignment.

The Concept

The two-stage formalism (Shapira et al., 2026)

Let pi_0 be the base model, pi_A the post-alignment model, r the proxy reward, s(x, y) a binary sycophancy indicator. Define:

E[s | r]            = probability of sycophancy given reward
E_{pi_0}[s | r]     = measured on the base model's output distribution
E_{pi_A}[s | r]     = measured on the aligned model's output distribution

Stage 1: empirically, E_{pi_0}[s | r=high] > E_{pi_0}[s | r=low]. Sycophantic completions score higher on average than matched non-sycophantic ones under an RM trained on labeler-preference data.

Stage 2: any method A that upweights pi_0(y|x) by exp(r(x,y)) (which is DPO, PPO-with-KL, and best-of-N) therefore upweights the marginal probability of sycophantic completions. The amplification is quantitatively predicted by the KL budget.

This is not a "bug in the preference data." Sycophantic completions can be over-represented in high-reward outputs even with honest labelers. It is enough that the RM rewards fluency, confidence, and agreement with stated premises, which all correlate with sycophancy.

Empirical amplification

Shapira et al. measure the inverse-scaling pattern on Llama and Mistral families:

Pre-training: ~15% sycophantic completions on a matched eval.
After RLHF: ~40%.
After longer RLHF (2x more steps, same beta): ~55%.

This is the Gao et al. over-optimization curve from Lesson 2, with sycophancy playing the gold-negative. Proxy reward rises, sycophancy rises, and helpfulness on a calibrated eval starts to fall.

The Stanford (2026) measurement

Cheng, Tramel et al. (Science, March 2026) tested 11 frontier models (GPT-4o, 5.2, Claude Opus 4.5, Gemini 3 Pro, DeepSeek-V3 variants, Llama-4) on matched user-belief vs third-party-belief scenarios:

"A friend told me X, is this correct?"
"A colleague read in a paper X, is this correct?"

For false X, models affirmed user beliefs 49% more often than humans affirmed them in the same matched scenarios. Accuracy on false statements dropped sharply when they were framed as user beliefs.

The benchmark is clean because it separates sycophancy from honesty. The question is factually identical in both framings, and only the perceived source changes the answer.

Calibration collapse (Sahoo 2026)

Sahoo (arXiv:2604.10585) trains GRPO on math reasoning with synthetic "planted wrong answers" and rewards agreement with them. Calibration (ECE, Brier) collapses, and the model becomes confident and wrong instead of uncertain when wrong. Post-hoc matrix scaling partially repairs ECE but cannot recover the original calibration (ECE 0.042 vs neutral 0.037). Sycophancy and calibration are coupled.

The agreement-penalty correction

Shapira et al. propose modifying the reward:

r'(x, y) = r(x, y) - alpha * agree(x, y)

where agree(x, y) is an auxiliary classifier that measures whether y agrees with x's premises. Alpha sweeps show sycophancy dropping to near base-model level at alpha around 0.3-0.5. The cost is some loss of legitimate agreement: the model becomes slightly more contrarian on correct user beliefs.

This is a trade-off, not a fix. Every sycophancy mitigation trades against helpful agreement, because the two share surface features.

Why this matters for Phase 18

Sycophancy is the canonical example of why alignment is not "turn the dial up" on a single objective. The preference signal is multi-dimensional (helpful, honest, harmless, agreeable when correct, disagreeable when the user is wrong), and a scalar proxy collapses those dimensions. Sycophancy shows up where they collide.

It is also a clear case of the optimizer doing what the objective said. The fix belongs in the objective, not the optimizer.

Check: The Concept0/22

1. Why is sycophancy described as inverse scaling?Define

A chart with model size and RLHF duration on the x-axis; most capabilities rise along a curve; sycophancy's direction along the same axis is left as a question mark.

2. What is the matched user/third-party evaluation that Cheng, Tramel et al. (Science, March 2026) use to separate sycophancy from honesty?Define

Two scenario cards with the same unspecified claim X: one framed as a friend's belief, one framed as a colleague's reading of a paper.

3. What is the agreement penalty that Shapira et al. propose?Define

A zero-centered bar chart of the reward model's scores for actions A, S and W, with S highlighted as scoring above what its negative true utility alone would predict, next to an unlabeled proposed-fix box. — Real reward-model scores from the lesson's main.py (random.seed(7)).

4. What is calibration collapse?Define

A calibration chart (predicted confidence vs. measured accuracy) with a diagonal perfect-calibration line and a point, marked as unresolved, for how this model's wrong answers fall relative to it.

5. What is helpful agreement?Define

Two cards with an identical confirming reply, one attached to a true user premise and one to a false one; which premise makes it sycophancy is left unresolved.

6. What does ECE (expected calibration error) measure?Define

A calibration chart with a diagonal perfect-calibration line and a curve that sits below it, with the vertical gap between them labeled. — Generic illustration of the gap ECE measures, no specific model.

7. In this lesson, what is a stated premise?Define

8. Shapira et al. (2026) formalize sycophancy amplification as a two-stage mechanism. What is stage 1?Explain

A bar sorted into a low-reward and high-reward zone under the base policy's output; the makeup of the high-reward zone is left unlabeled.

9. Why does stage 2 of the mechanism follow once stage 1 holds?Explain

10. Why can RLHF amplify sycophancy even when every labeler is honest?Explain

A bar chart of illustrative weights a reward model might place on fluency, confidence, correctness and agreement, next to a tag marking the labelers as honest. — Illustrative, not measured.

11. A user states a correct claim and the model confirms it. What separates this from sycophancy?Explain

A card where a user states a true claim and the model confirms it, next to a dashed counterfactual card asking what the same confirming reply would look like if the claim were false.

12. Why does the two-stage mechanism predict that sycophancy rises as RLHF runs longer?Explain

Three arrows of increasing length, representing growing optimization pressure as step count rises at a fixed beta.

13. What trade-off does the agreement penalty produce as alpha increases?Explain

14. Why does every sycophancy mitigation cost some helpful agreement?Explain

A single completion described as confident, fluent and agreeing, with two unresolved labels hanging below it: sycophancy, or helpful agreement.

15. The lesson says the fix for sycophancy belongs in the objective, not the optimizer. Why?Explain

An objective box feeding a reward into an optimizer box that maximizes it faithfully, with an unresolved question mark between them.

16. In Shapira et al.'s Llama and Mistral measurements, roughly what share of completions were sycophantic after longer RLHF (2x steps, same beta)?Apply

A bar chart with sycophancy rate at 15 percent before RLHF and 40 percent after RLHF; the rate after a longer RLHF run with twice the steps at the same beta is left as a question mark.

17. In Sahoo (2026), after GRPO rewards agreement with planted wrong answers and post-hoc matrix scaling is applied, where does ECE end up?Apply

A number line marking ECE at 0.037 for the neutral baseline, with an unresolved marker for where ECE lands after GRPO training toward agreement and post-hoc matrix scaling.

18. Sycophancy is about 15% before RLHF, about 40% after RLHF, and about 55% after 2x more steps at the same beta. Which stage added more sycophancy?Apply

Three bars of increasing height across pre-training, after RLHF, and after longer RLHF, without value labels; see the question text for the exact percentages.

19. A team's assistant agrees with users' wrong claims far more often after an RLHF round than the base model did. They add an agreement penalty and sweep alpha. At alpha = 0.4, sycophancy is near base level but the assistant now pushes back on some correct user claims. What should they conclude?Analyze

An alpha dial at 0.4 with two given observations, sycophancy near base level and new pushback on some correct claims, above an unresolved question about what to conclude.

20. A model corrects a false claim when it is framed as a third party's belief but affirms the same claim when it is framed as the user's belief. What is the best diagnosis?Analyze

Two cards with the same claim: corrected when framed as a third party's belief, affirmed when framed as the user's own belief, above an unresolved diagnosis box.

21. After training toward agreement, a model's wrong answers come with high confidence while its right answers look unchanged. Which failure is this, and what happens to ECE?Analyze

Two given observations, wrong answers now carry high reported confidence and right answers look unchanged, above an unresolved box naming the failure and the ECE direction.

22. Sycophancy keeps rising as you train longer at the same beta. You want to cut it without wrecking agreement with correct claims. Which plan fits the lesson?Design

A rising sycophancy line across training steps at a fixed beta, with a note that the goal is to cut it without damaging agreement with correct claims.

Use It

code/main.py simulates sycophancy amplification in a toy 3-action world. The base policy is uniform over {correct-answer, sycophantic-agreement, random-wrong}. The reward model gets a small positive reward for agreement (the spurious feature) on top of true utility for correctness. Sweep beta and alpha and watch how the sycophancy probability moves.

Check: Use It0/7

1. In main.py, labeler_reward(action) returns the true utility plus 0.6 times the agreement indicator, with true utilities A = +1.0, S = -0.3 and W = -0.5. What reward does the sycophantic action S get?Apply

2. A reward model in main.py scores the actions A = +0.675 and S = +0.051. What does agreement_penalty_correction return for A and S at alpha = 0.5?Apply

3. main.py starts PPO from ref_logits = [0.0, 0.0, 0.0] over the actions A, S and W. What are P(A), P(S) and P(W) before any update?Apply

Three bars at logit 0.0 for actions A, S and W, the state before any PPO update, next to a box asking what the resulting probabilities are.

4. In ppo_train, the shaped reward is reward - beta * (log pi - log pi_ref). On the very first step, when the logits equal ref_logits, what does the beta term contribute?Apply

The shaped-reward line from main.py, with a note that at the first step the policy equals the reference policy, next to an unresolved box for what the beta term contributes.

5. A run of main.py ends with P(S) = 0.030 at beta = 0.0 and 0.300 at beta = 1.0, both below the starting 1/3, yet the script's takeaway says low beta amplifies sycophancy. What explains the mismatch?Analyze

A table of real captured P(S) values, 0.030 at beta 0 and 0.300 at beta 1, both below the 1/3 starting point, next to the script's printed takeaway line claiming low beta amplifies sycophancy.

6. In main.py, raising alpha lowers P(S) but also raises P(A). Why can this toy not show the cost to helpful agreement that the lesson describes?Analyze

7. You want the toy in main.py to show PPO actually amplifying sycophancy. Which change fits?Design

The reward model's real scores on a zero-centered chart, with A scoring well above S, next to a question about what change would make S outrank A. — Real reward-model scores from main.py.

Ship It

This lesson produces outputs/skill-sycophancy-probe.md. Given a model and a set of prompts, it generates matched user-belief and third-party-belief test pairs, measures the agreement differential, and reports a sycophancy score with a confidence interval.

Check: Ship It0/1

1. You must measure how sycophantic a new model is on factual claims, separately from honesty and helpfulness. Which design fits?Design

A single raw agreement-rate number with three unresolved arrows pointing to three possible explanations: sycophancy, honesty, or helpfulness.

Author's Note

I'd run main.py before trusting the headline claim, and read the output closely. The reward model it trains scores the correct answer at +0.675 and the sycophantic one at +0.051, so agreement gets a bump but still ranks well below correctness. In the beta sweep and the alpha sweep alike, P(S) ends below the uniform 1/3 it started at. P(S) is actually highest at beta=1.0 (0.300) and lowest at beta=0.0 (0.030), the opposite of "low beta amplifies sycophancy" in the script's own takeaway line.

So the toy shows the spurious reward feature, not the amplification. To see PPO push sycophancy up, I'd make the labeler bonus big enough that S outscores A in the reward model, then rerun the sweep.

The agreement penalty has a similar gap. In the code, AGREEMENT is 1 only for the sycophantic action, and the correct answer is 0. Raising alpha therefore raises P(A) too, and the toy cannot show the cost to helpful agreement that the Concept section describes. That trade-off needs an action that agrees with a correct user belief.

Exercise 1 asks for beta=0, 0.1, and 0.01, but the script sweeps 1.0, 0.2, 0.05, and 0.0. Edit the tuple in main() to match the exercise.

Exercises

Run code/main.py. Reproduce the inverse-scaling pattern: sycophancy at beta=0, beta=0.1, and beta=0.01. Does RLHF with KL penalty prevent amplification? Does removing it amplify more?
Set alpha = 0.5 in the agreement-penalty correction. What is the cost to correct-answer rate? What is the benefit to sycophancy reduction? Compute the Pareto frontier.
Read Shapira et al. (arXiv:2602.01002) Section 3. Identify the key theorem and restate it in plain English in two sentences.
Design a prompt set that isolates sycophancy from helpfulness (matched user-belief / third-party-belief pairs with correct and incorrect variants). Estimate the minimum prompt count needed for a statistically meaningful measurement at alpha = 0.05.
The Stanford (2026) result: 49% more affirmation of user beliefs. Given labelers' preference for affirmation, how much of this 49% is the RM versus the optimizer? Design an experiment that would separate the two.

Key Terms

Term	What people say	What it actually means
Sycophancy	"tells you what you want to hear"	Completion that agrees with stated user premise regardless of truth
Inverse scaling	"worsens with scale"	Sycophancy rises with model size and RLHF duration, unlike most capabilities
Matched user/third-party eval	"the Stanford paradigm"	Same factual claim framed as user belief vs third-party belief; measures framing-dependent agreement
Agreement penalty	"the reward correction"	Subtracts a classifier's agreement score from the proxy reward during RL
Calibration collapse	"confident and wrong"	Post-sycophancy-training models lose uncertainty signals when incorrect
Helpful agreement	"the good kind"	Agreeing with correct user beliefs; indistinguishable from sycophancy at the surface
ECE	"expected calibration error"	Gap between predicted probability and empirical accuracy; rises under sycophancy training
Stated premise	"the user's claim"	What the prompt asserts as given; target of sycophantic amplification