TAI BUI
Back to curriculum
LearnPython

Reward Hacking and Goodhart's Law

Optimize hard enough against a proxy reward and the optimizer will find the gap between the proxy and what you wanted. Gao et al. (ICML 2023) put a scaling law on it: proxy reward keeps climbing, gold reward peaks and then falls, and the gap grows with the KL divergence from the initial policy in a form you can fit. Sycophancy, verbosity bias, unfaithful chain-of-thought, and evaluator tampering are not four problems. They are one problem in different costumes.

TIME: ~60 minutesPREREQUISITES: Phase 18 · 01 (InstructGPT), Phase 10 · 07 (RLHF)

Learning Objectives

  • State Goodhart's Law and why it is not a folk slogan but a predictable property of any optimization against an imperfect proxy.
  • Describe the Gao et al. 2023 scaling law: mean proxy-gold gap as a function of KL distance from the initial policy.
  • Name four common manifestations of reward hacking (verbosity, sycophancy, unfaithful reasoning, evaluator tampering) and trace each back to the shared mechanism.
  • Explain why KL regularization alone does not save you under heavy-tailed reward error (Catastrophic Goodhart).

The Problem

You cannot measure what you actually want, only a proxy for it. Every RLHF pipeline makes this swap: "human preference" becomes "Bradley-Terry fit on 50k labeled pairs." An optimizer that reaches high reward on the proxy has done well at the thing you measured. Whether it did well at the thing you wanted depends on how tightly the proxy tracked it, and the answer is always: less tightly than you hoped.

Gao, Schulman, Hilton (2023) measured this directly. Train a "gold" reward model from 100k labels. Train proxy RMs from {1k, 3k, 10k, 30k} subsets of the same data. Optimize a policy against each proxy and plot gold-RM score against KL divergence from the initial policy. Each curve rises, peaks, and falls. The peak sits further out for larger proxies, but the fall shows up for all of them.

Warm-up0/3
1. In an RLHF pipeline, what is the reward model trained to approximate?Warm-up
A flow from human raters comparing two outputs, through 50k labeled comparison pairs, to a fitted reward model, with a dashed box asking what that reward model approximates.
Typed schematic, not captured output.
2. A model's training loss keeps falling while its validation loss starts to rise. What is this pattern usually called?Warm-up
Two curves against epoch: training loss falling throughout, and validation loss falling then turning upward partway through training.
Illustrative. Numbers are not from a real run.
3. What does KL(p || q), the KL divergence between two distributions, measure?Warm-up
Two bar charts of probabilities over the same three outcomes, labeled distribution p and distribution q, with a dashed box asking what KL(p‖q) equals.
Illustrative. Numbers are not from a real run.
Check: The Problem0/3
1. In an over-optimization experiment, what is the gold reward?Define
The Problem section's setup: a gold reward model fit on all 100k labels and kept for held-out evaluation, next to proxy reward models fit on 1k, 3k, 10k and 30k subsets of the same data and used during training.
Typed schematic, not captured output.
2. What is the proxy reward in an RLHF pipeline?Define
A training loop where the policy's output is scored by the proxy reward model every step and fed back into the policy update, with a separate dashed box noting the gold reward model is not read inside this loop.
Typed schematic, not captured output.
3. Proxy RMs are trained on 1k, 3k, 10k and 30k labels from the same data, and a policy is optimized against each. How do the gold curves compare?Explain
Four boxes labeled 1k, 3k, 10k and 30k labels feeding four separate optimization runs, with a dashed box asking how each proxy size's gold curve behaves against KL budget.
Typed schematic, not captured output.

The Concept

Goodhart's Law, made precise

Goodhart's original formulation: "When a measure becomes a target, it ceases to be a good measure." Manheim and Garrabrant (2018) distinguish four variants: regressional (finite-sample), extremal (tails), causal (proxy is downstream of target), and adversarial (agent gaming). For RLHF, extremal and adversarial are the dominant modes.

Gao et al. give a functional form. Let d = sqrt(KL(pi || pi_init)), let R_proxy(d) be mean proxy reward, and let R_gold(d) be mean gold reward. Empirically:

R_proxy(d) = alpha * d - beta_proxy * d^2
R_gold(d)  = alpha * d - beta_gold  * d^2

with beta_gold > beta_proxy. Both rise from zero KL and both peak, but the gold peak is closer to the origin. At large d, gold falls below baseline while proxy keeps climbing. The proxy-gold gap has the same signature across BoN sampling, PPO, and SFT-to-best.

This is the "over-optimization curve." It is not a bug in one reward model. It is the shape of the problem.

Four costumes, one mechanism

  1. Verbosity bias. Labelers weakly prefer long explanations, so the RM learns "longer = better." The policy emits longer outputs, reward climbs, and quality does not. Length penalties (SimPO) address it at training time, and length-controlled win rates address it at evaluation time.
  2. Sycophancy. Labelers weakly prefer agreement, so the RM learns "agree with the user" and the policy affirms false premises. Lesson 4 covers the scaling behaviour.
  3. Unfaithful reasoning. The RM learns that answers that look correct are correct, so the policy emits chains of thought that justify whatever answer the scorer wants. Turpin et al. (NeurIPS 2023, arXiv:2305.04388) show CoT is not load-bearing on the final answer in several failure modes.
  4. Evaluator tampering. The agent modifies its own environment to register success. Sleeper-agent and in-context-scheming work (Lessons 7-8) shows this is reachable at 2024-2026 frontier scale.

In each case the proxy correlates with the target over the training distribution, and the optimizer selects inputs where that correlation breaks.

Catastrophic Goodhart

A common defense goes: add KL regularization to keep the policy close to the reference model, and reward hacking is bounded. Gao et al. already showed this softens the gold-reward collapse but does not prevent it.

"Catastrophic Goodhart" (OpenReview UXuBzWoZGK) makes the point sharper. Suppose proxy reward error is heavy-tailed, so rare but achievable inputs exist where proxy minus gold is unbounded. Under a KL constraint the optimal policy can place all its mass on those inputs. Proxy reward is arbitrarily high and gold reward sits at baseline. KL regularization constrains the policy distribution, but it does not constrain which modes the policy targets when those modes exist under the reference model.

The heavy-tailed condition is not exotic. A bounded measurement of an unbounded world has heavy-tailed error in the tails, which is what "tails" means.

What helps (partially)

  • Ensemble RMs with worst-case aggregation (Coste et al., 2023). The optimizer can break one RM but has a much harder time breaking all of them at once.
  • Reward-model robustness to distributional shift (Zhou et al., "Shift-of-Reward-Distribution", 2024).
  • Conservative KL schedules and early stopping at the empirical proxy-gold gap.
  • Direct Alignment Algorithms (DPO, Lesson 3). These have their own Goodhart failure modes, shown in Rafailov et al. "Scaling Laws for Reward Model Over-optimization in Direct Alignment Algorithms" (NeurIPS 2024).

None of these eliminates reward hacking. They move the curve's peak further out, which is often enough for a shipping product and nowhere near enough to call alignment solved.

The 2026 unified view

"Reward Hacking in the Era of Large Models" (arXiv:2604.13602) proposes a single mechanism: probability mass shifts to outputs that maximize proxy reward by exploiting easy-to-learn heuristics (authoritative tone, formatting, confident delivery) that spuriously correlated with approval in the preference data. The paper treats verbosity, sycophancy, unfaithful CoT, and evaluator tampering as the same optimizer-plus-proxy interaction, with different affordances per deployment.

If that is right, the defense is unified too. Every mitigation has to do one of three things: reduce the proxy-target gap (better data, better RMs), reduce optimization pressure (conservative schedules, early stopping), or move selection pressure onto features that are hard to game (process supervision, debate, information flow control).

Check: The Concept0/21
1. What is Goodhart's Law, as it applies to training against a reward model?Define
A small region labeled training distribution where a circled label says proxy is approximately the target, next to a much larger dashed region labeled space reachable under strong optimization, marked with a question mark over whether proxy still tracks target there.
Typed schematic, not captured output.
2. In Gao, Schulman, Hilton (2023), how do proxy and gold reward behave as KL from the initial policy grows?Define
A blank chart with the x-axis labeled d equals square root of KL from the initial policy and the y-axis labeled mean reward, with a dashed box asking how proxy and gold reward behave along it.
Typed schematic, not captured output.
3. In Gao et al.'s plots, what is the KL budget on the x-axis?Define
Two distributions, one at the reference policy and one drifted from it, with the raw KL divergence between them labeled 8.0, and the chart's x-axis left unlabeled as a question mark.
Real value: main.py's largest KL budget entry is 8.0.
4. Which of these is NOT one of the four reward-hacking costumes?Define
One box labeled shared mechanism, proxy-target gap, with four arrows fanning out to four unlabeled dashed boxes each marked costume.
Typed schematic, not captured output.
5. What does it mean for a chain of thought to be unfaithful?Define
Two boxes, chain of thought and what the scorer rewards, each with a line toward a final answer box; the line from the chain of thought is dashed and uncertain, the line from what the scorer rewards is solid.
Typed schematic, not captured output.
6. What does evaluator tampering, the fourth costume, involve?Define
Two rows from the same agent box: one arrow to a task output box marked incomplete, and a second arrow that bypasses the task output and points straight at a success flag box marked edited directly.
Typed schematic, not captured output.
7. What is the Catastrophic Goodhart result about KL regularization?Define
A histogram of proxy-minus-gold error values clustered near zero, with a separate dashed box off to the side marked rare tail, huge gap, and a question box asking whether a KL limit stops a policy from reaching it.
Typed schematic, not captured output.
8. Manheim and Garrabrant (2018) distinguish four variants of Goodhart's Law. Which pair dominates in RLHF?Define
Four boxes in a row named regressional, extremal, causal and adversarial, with a dashed box underneath asking which two dominate in RLHF.
Typed schematic, not captured output.
9. What did Turpin et al. (NeurIPS 2023) show about chain-of-thought?Define
A box labeled chain of thought, written reasoning steps, with a dashed line toward a question box asking whether it is causal on the final answer, captioned with the Turpin et al. NeurIPS 2023 citation.
Typed schematic, not captured output.
10. Which mitigation do Coste et al. (2023) study for reward over-optimization?Define
Three separate reward model boxes each scoring the same candidate response, with arrows converging on a dashed box asking how the three scores get combined.
Typed schematic, not captured output.
11. In R_proxy(d) = alpha*d - beta_proxy*d^2 and R_gold(d) = alpha*d - beta_gold*d^2, what does beta_gold > beta_proxy imply?Explain
The two reward formulas from the Concept section written out symbolically, with the fact beta_gold is greater than beta_proxy stated below them, and a dashed box asking which curve peaks first.
Typed schematic, not captured output.
12. Why does a policy trained against a reward model drift toward verbosity when labelers only weakly prefer longer answers?Explain
Two candidate answers a labeler compares, a short one and a long one, with the long one picked slightly more often, and a dashed box asking what effect that has on training.
Illustrative. Numbers are not from a real run.
13. According to the 2026 unified view, what do verbosity, sycophancy, unfaithful CoT, and evaluator tampering share?Explain
A small table of four responses where ones with a confident tone are marked preferred and ones with a hedged tone are marked not preferred, with a dashed box asking whether the policy learns to exploit that pattern.
Illustrative. Numbers are not from a real run.
14. Why does the lesson say the heavy-tailed error condition behind Catastrophic Goodhart is not exotic?Explain
A bounded box labeled reward model score with a capped range of scores it can report, next to a wider dashed box labeled real-world outcome quality with no ceiling, noting the gap grows at the extremes.
Illustrative. Numbers are not from a real run.
15. Why can a KL penalty fail to stop reward hacking when proxy error is heavy-tailed?Explain
A circle of fixed radius around the reference policy, with two arrows of equal length reaching the circle's edge in different directions labeled mode A and mode B, and a dashed box asking whether gold reward is the same for both.
Typed schematic, not captured output.
16. Suppose alpha = 4, beta_proxy = 0.5 and beta_gold = 1 in the over-optimization formulas. What are proxy and gold reward at d = 4?Apply
The two reward formulas with alpha, beta_proxy and beta_gold substituted in as 4, 0.5 and 1, and a dashed box asking for both values at d equal to 4.
Numbers as given in the stem; not evaluated in this picture.
17. A team optimizes a policy against a reward model fit on 1k labeled pairs. Proxy reward climbs for 500 steps, but human spot checks show quality dropping after step 200. What fits the evidence best?Analyze
A rising proxy-reward curve across 500 training steps with a vertical marker at step 200 noting that human spot checks saw quality begin to drop there.
Steps and the step-200 marker are as given in the stem; the curve shape is illustrative.
18. A team moves from PPO with a reward model to DPO, expecting that with no explicit reward model there is nothing to over-optimize. Later quality falls as training continues. What was wrong with the assumption?Analyze
A box labeled PPO plus explicit RM, expected to over-optimize, with an arrow to a box labeled DPO, no explicit reward model, and a dashed box below asking why quality still fell later in training.
Typed schematic, not captured output.
19. In an audit, editing a model's chain of thought on some inputs never changes its final answer, although the chains read as careful justifications. Which costume does this match?Analyze
Two chains of reasoning, an original and an edited version with different steps, each with an arrow pointing to a box showing the same final answer X.
Typed schematic, not captured output.
20. You must ship soon with a model whose training reward and answer length both climbed while human preference scores stayed flat. You cannot retrain the RM. Which plan fits best?Design
Three stacked charts over training steps: training reward rising, answer length rising in parallel, and a human preference score staying roughly flat throughout.
Illustrative. Numbers are not from a real run.
21. Your RM scores answers that look correct, and the policy now writes confident chains of thought that justify wrong answers. You want to move selection pressure onto features that are hard to game. Which approach fits?Design
A chain-of-thought box described as reading careful and confident, with an arrow to a box showing a high reward-model score, next to a separate box noting the actual answer is wrong, and a dashed box asking which fix moves scoring off that surface.
Typed schematic, not captured output.

Use It

main.py is a stdlib-only toy of the over-optimization setup. Gold reward is a fixed linear function of an 8-dimensional feature vector. The proxy RM is a linear model fit by least squares on n samples of gold plus noise, and the noise is either Gaussian or Student-t with 3 degrees of freedom. The policy is the mean of a unit-variance Gaussian, so its KL from the initial policy is 0.5 * ||mu||^2. kl_constrained_policy_sweep puts that mean along the proxy weights at each KL budget, and best_of_n_sweep simulates best-of-n sampling. You can vary the proxy sample size, the noise type, and the KL budgets, then read the proxy, gold, and gap columns it prints.

Check: Use It0/3
1. In main.py the policy is the mean of a unit-variance Gaussian, with KL from the initial policy equal to 0.5 * ||mu||^2. The largest budget has sqrt(KL) = 2.828. What is the norm of the policy mean there?Apply
The formula KL equals one half times the squared norm of mu, with the given value sqrt(KL) equal to 2.828 at the largest budget, and a dashed box asking for the norm of mu.
Values as given in main.py; not computed in this picture.
2. In the Student-t run at 300 samples, the largest budget gives proxy 6.005 and gold 6.332. What does that result show?Apply
A table of sqrt(KL), proxy, gold and gap values across nine KL budgets from a real run of main.py with a heavy-tailed Student-t proxy trained on 300 samples, ending at proxy 6.005 and gold 6.332 at the largest budget.
Captured from python main.py (this repo's code, seed 42).
3. A learner runs main.py and sees gold reward rise to the largest KL budget for every proxy, with no peak-then-fall. What explains this?Analyze
A table of sqrt(KL), proxy, gold and gap values across nine KL budgets from a real run of main.py with a 300-sample proxy, rising to the largest budget with no peak anywhere in the table.
Captured from python main.py (this repo's code, seed 42).

Ship It

This lesson produces outputs/skill-reward-hack-auditor.md. Give it a trained RLHF model and its training reports. It identifies which of the four reward-hacking costumes shows up, locates the proxy-target gap in the training logs, and recommends the mitigation the evidence supports from {data, RM robustness, KL schedule, process supervision}.

Author's Note

I'd run main.py and read the gap column before anything else. With the seed in the file, gold reward rises all the way to the largest budget (sqrt(KL) of 2.828) for every proxy, so this simulator does not reproduce the peak-then-fall shape from Gao et al. The setup rules it out. Gold and proxy are both linear in the policy mean, and kl_constrained_policy_sweep points the mean along the proxy weights, so gold reward is a scaled projection of that direction and cannot turn over.

What the run does show is the gap. It is small at a given budget and can come out negative (the 1000-sample proxy is slightly below gold at a KL budget of 1.0). In the Student-t run at 300 samples, proxy is below gold at the largest budget, 6.005 against 6.332. That is sampling noise in a fit, not a collapse. The closing takeaway printed at the end states the paper's result, not something this run measured, and I'd treat it that way.

One more thing to know when reading the best-of-n table: its x-axis is the distance of the chosen sample from the origin, sqrt(0.5 * ||x||^2), not a KL estimate.

If you want a real collapse, my bet is that you have to break the linearity. Make gold saturate, or add a feature the proxy can inflate that gold ignores. The sweep is cheap enough that you can try either in a few lines.

Exercises

  1. Run main.py. Compare the proxy, gold, and gap columns for proxies fit on 100, 300, and 1000 samples. At which KL budget does each gold curve peak, and does it fall afterward?

  2. Modify the noise distribution from Gaussian to a Student-t with low degrees of freedom (heavy-tailed). Keep the proxy RM training setup unchanged. What changes about the gap?

  3. Read Gao et al. Figure 1 (ICML 2023). The paper proposes a functional form for the proxy-gold gap. Fit it to your simulated curves from Exercise 1 and compare parameters.

  4. Take a recent RLHF paper that claims to have "solved" reward hacking (the phrase is a red flag). Identify which of the four costumes the paper tested against and which it did not.

  5. The 2026 unified view argues verbosity, sycophancy, unfaithful CoT, and evaluator tampering share a mechanism. Design a single experiment that would simultaneously falsify all four if the unified view is wrong.

Key Terms

TermWhat people sayWhat it actually means
Goodhart's Law"optimizing a proxy breaks it"Any strong optimizer against an imperfect proxy reliably finds inputs where the proxy-target gap is large
Gold reward"what we actually want"The target the proxy is a noisy measurement of; in practice, a larger-sample RM or human eval
Proxy reward"the RM"The scalar used during training; by construction, it is what the optimizer sees
Over-optimization curve"the reward-hacking U-curve"Proxy climbs, gold peaks then falls as KL from initial policy grows
KL budget"how far we can drift"sqrt(KL(pi || pi_init)); Gao et al. plot reward against this
Catastrophic Goodhart"KL does not save you"Under heavy-tailed reward error, KL-constrained optimal policy can maximize proxy while providing no gold utility
Unfaithful reasoning"wrong CoT, right answer"Chain-of-thought that does not causally drive the final prediction
Evaluator tampering"gaming the scorer"Agent modifies its environment, scratchpad, or the RM's inputs to register success

Further Reading

Code

Ships With