LearnPython

Scalable Oversight and Weak-to-Strong Generalization

Burns et al. (OpenAI Superalignment, "Weak-to-Strong Generalization", 2023) proposed a proxy for the superalignment problem: fine-tune a strong model on labels produced by a weaker one. If the strong model generalizes correctly from imperfect weak supervision, current human-scale alignment methods may extend to superhuman systems. Scalable oversight (debate, recursive reward modeling, task decomposition) raises the overseer's effective capability so it can keep up with the model. W2SG makes sure the strong model generalizes correctly from whatever imperfect supervision the overseer provides. Debate Helps W2SG (arXiv:2501.13124, January 2025) combines the two.

TIME: ~60 minutesPREREQUISITES: Phase 18 · 01 (instruction-following), Phase 18 · 10 (AI Control), Phase 09 (RL foundations)

Learning Objectives

Define scalable oversight and weak-to-strong generalization and explain how they are complementary.
Describe the Burns et al. 2023 experimental setup: fine-tune GPT-4 using labels from GPT-2.
Explain the performance gap recovered (PGR) metric and what it measures.
State the three major scalable-oversight mechanisms (debate, recursive reward modeling, task decomposition) and one strength of each.

The Problem

Every alignment technique so far in Phase 18 assumes the overseer can evaluate the model's behaviour. When the model is superhuman, the overseer is the weak link. The superalignment question: can a weaker supervisor reliably produce a stronger, aligned model?

Burns et al. turn this into an empirical setup. Supervise a strong model with a weak one, then measure how much of the strong model's capability survives. This does not solve superalignment. It is a proxy that lets you measure progress.

Warm-up0/3

1. In RLHF, what does the reward model learn to do?Warm-up

A flow from human labelers comparing two outputs, through preference data on which one is better, to a fitted reward model, with a dashed box asking what the reward model's output is. — Typed schematic, not captured output.

2. A defensive AI-control approach to safety assumes what about the model being deployed?Warm-up

A pipeline where a model proposes an action, a monitor and action gate check it before it reaches the environment, with a dashed box asking what this setup assumes about the model. — Typed schematic, not captured output.

3. When people evaluate a classifier, what is a 'gold label'?Warm-up

The same example gets two labels: three trained annotators agree it is spam and that becomes the gold label, while a small model's own quick guess on the same example says not spam. — Typed schematic, not captured output.

Check: The Problem0/1

1. What does 'superalignment' refer to?Define

A quoted sentence asking whether a weaker supervisor can reliably produce a stronger, aligned model, with a dashed box below asking what this question is called. — Quoted directly from the Problem section.

The Concept

W2SG: the Burns et al. setup

Weak model: GPT-2 class.
Strong model: GPT-4 class.
Target: strong GPT-4 ceiling on the task.

Procedure:

Get weak model's zero-shot predictions on a task.
Fine-tune the strong model on weak-labeled data.
Measure the strong fine-tuned model's accuracy.

Baselines: weak model's own accuracy, strong model's supervised ceiling on gold labels.

Gap metric: Performance Gap Recovered (PGR) = (fine-tuned - weak) / (ceiling - weak). PGR of 1.0 means weak supervision fully closes the gap; PGR of 0 means weak supervision did not help.

Burns et al. empirical findings

Across NLP, chess-puzzle, and reward-modeling tasks, PGR was consistently positive: the strong model generalized beyond the weak supervisor's mistakes. Task-specific PGR varied from ~20% to ~80%. The strong model "understands" the intended task from imperfect labels and fills in with its pre-trained priors.

Limitations Burns et al. note:

The weak-to-strong gap here is capability, not alignment. The weak model is aligned, while a truly superhuman model might be overseen by an aligned human with a much larger capability gap.
The strong model's generalization may be more about task priors than about recovering ground-truth intent.
Eliciting the strong model's latent knowledge is the real problem; PGR measures a specific operationalization.

Scalable oversight: three mechanisms

Debate (Irving et al. 2018). Two instances of U argue opposite sides; the weak judge decides. The assumption: finding a convincing true answer is easier than finding a convincing false answer. Recent 2024-2025 work (Khan et al., Michael et al.) finds that debate sometimes helps and sometimes does not, depending on task structure.
Recursive Reward Modeling (Leike et al. 2018). U helps the human train the reward model for U+1. The overseer's effective capability grows with the model's.
Task Decomposition (Christiano, Shlegeris, Amodei 2018). Decompose a hard task into sub-tasks the human can check, recursively. Assumes decomposability.

Each mechanism assumes something about the structure of the task or the alignment of the intermediate components.

Why scalable oversight and W2SG are complementary

Scalable oversight raises the overseer's effective signal quality. W2SG closes the gap from whatever imperfect signal the overseer can still provide.

Lang et al., Debate Helps Weak-to-Strong Generalization (arXiv:2501.13124), combine them. A debate protocol provides better weak labels, and the strong model is trained on those labels. They report PGR gains on NLP tasks.

The organizational drama

OpenAI's Superalignment team dissolved in May 2024 after Jan Leike's departure to Anthropic. The agenda (scalable oversight, W2SG, automated alignment research) continued at Anthropic and at academic labs: MATS (Lesson 28), Redwood (Lesson 10), Apollo (Lesson 8), METR (Lesson 28). The organization changed and the research questions stayed.

Where this fits in Phase 18

Lessons 6-10 cover the threat and the defensive paradigm, which assumes U is untrustworthy. This lesson is the offensive paradigm: make the overseer strong enough to verify U's alignment. Lessons 12-16 then turn to the practical tooling of adversarial evaluation.

Check: The Concept0/22

1. What does scalable oversight try to achieve?Define

Two small bar charts: before a mechanism, the overseer's bar is much shorter than the model's; after an oversight mechanism, the overseer's bar is closer to the model's height. — Typed schematic, not captured output.

2. What is weak-to-strong generalization (W2SG)?Define

A weak model's zero-shot labels feeding into a strong model that gets fine-tuned on them, with a dashed box asking what the fine-tuned strong model then gets measured against. — Typed schematic, not captured output.

3. How is Performance Gap Recovered (PGR) defined?Define

A number line with three illustrative markers labeled weak, fine-tuned and ceiling at different points, and a dashed box asking for the PGR formula that uses those three values. — Illustrative. Numbers are not from a real run. Positions only; no formula applied here.

4. In debate as a scalable-oversight mechanism, how is the answer decided?Define

One question with arrows to two debaters, one arguing yes and one arguing no, and a dashed box between them for a judge asking how the judge picks. — Typed schematic, not captured output.

5. What is Recursive Reward Modeling (RRM), as proposed by Leike et al. in 2018?Define

A human and model U both feeding into a dashed box asking what they train together. — Typed schematic, not captured output.

6. What is task decomposition in scalable oversight?Define

One box labeled hard task with arrows fanning out to three boxes each labeled sub-task, and a dashed box below asking who checks each sub-task and how often that splitting repeats. — Typed schematic, not captured output.

7. Which of the following is NOT one of the three scalable-oversight mechanisms the lesson names?Define

Four equally styled boxes named debate, tokenizer distillation, recursive reward modeling and task decomposition, with none of them marked or highlighted. — All four named exactly as in the options; none marked.

8. In Burns et al. (2023) weak-to-strong generalization, what is the setup?Explain

Two boxes named GPT-2 class and GPT-4 class, each with a dashed box underneath asking what role it plays in the setup. — Model classes as named in the Concept section; roles left blank.

9. Why does the PGR calculation need both a weak baseline and a gold-label ceiling?Explain

Two number lines: the first has only a weak marker with a question about how much is recovered from one point alone, the second adds fine-tuned and ceiling markers to the same scale. — Illustrative. Numbers are not from a real run. Positions only; no formula applied here.

10. Across the NLP, chess-puzzle and reward-modeling tasks, what did Burns et al. report for PGR?Explain

Three boxes named NLP, chess puzzle and reward modeling, each asking what PGR was measured on that task family, with no values filled in. — The lesson states a roughly 20% to 80% range by task; exact per-task figures are not given.

11. What explains how a strong model can beat the weak labels it was fine-tuned on?Explain

Both facts as given in the Concept section.

12. Why does the lesson call the Burns et al. setup a proxy and not a solution to superalignment?Explain

A quoted sentence saying the setup does not solve superalignment and is a proxy that lets you measure progress, with a dashed box asking why it is only a proxy. — Quoted directly from the Problem section.

13. What is the central assumption of the debate mechanism?Explain

Two boxes, a true answer argued convincingly and a false answer argued convincingly, with a dashed box below asking which one is easier to produce. — Typed schematic, not captured output.

14. How does recursive reward modeling differ from debate in where the stronger oversight comes from?Explain

Two named boxes, recursive reward modeling and debate, each with its own dashed box underneath asking where its extra oversight strength comes from. — Typed schematic, not captured output.

15. How are scalable oversight and W2SG complementary?Explain

Two boxes, scalable oversight acting on the overseer's signal and W2SG acting on the strong model's training, with a dashed box below asking how the two combine. — Typed schematic, not captured output.

16. What limitation do Burns et al. note about their weak-to-strong setup?Explain

Two boxes, a GPT-2 class supervisor described as aligned but weaker than GPT-4, and a human overseeing a future superhuman model, with a dashed box asking whether the two face the same kind of gap. — Typed schematic, not captured output.

17. Which order matches the Burns et al. procedure?Apply

Three steps labeled A, B and C arranged in a triangle, not in their correct order: A is fine-tune the strong model on weak labels, B is measure the fine-tuned model's accuracy, C is collect the weak model's zero-shot predictions, with a dashed box asking for the correct order. — Typed schematic, not captured output.

18. A team fine-tunes a large model on labels from a small model. The small model alone scores 70%, the large model trained on gold labels scores 90%, and the large model trained on the small model's labels scores 80%. What is the PGR and what does it mean?Apply

A number line with three markers, weak at 70, fine-tuned at 80 and ceiling at 90, and a dashed box asking for PGR. — Values as given in the stem; not computed in this picture.

19. A weak model scores 60% and its strong counterpart scores 90% on gold labels. The strong model fine-tuned on weak labels reaches a PGR of 0.6. What accuracy did that fine-tuned model reach?Apply

A number line with weak at 60 and ceiling at 90, a note that PGR is 0.6 of the gap between them, and a dashed box asking for the fine-tuned accuracy. — Values as given in the stem; not computed in this picture.

20. A report says 'weak supervision recovered 80% of the gap, so a human can safely supervise a superhuman model.' Which flaw undermines this claim?Analyze

A quoted claim that weak supervision recovered 80% of the gap, so a human can safely supervise a superhuman model, with a dashed box asking what 80% PGR actually measures. — Claim quoted exactly as given in the stem.

21. A team wants to oversee a task whose pieces cannot be separated, because every sub-answer depends on the whole. Which mechanism's assumption fails?Analyze

A box stating the task's every sub-answer depends on the whole, above three equally styled boxes named debate, recursive reward modeling and task decomposition, with a note asking whose assumption breaks here. — Typed schematic, not captured output.

22. You only have a noisy weak labeler but want the strongest signal for fine-tuning a large model. Which design fits the lesson's ideas?Design

Two boxes, have: a noisy weak labeler and nothing else, and want: the strongest signal to fine-tune a large model, with a dashed box below asking for the best design using this lesson's ideas. — Typed schematic, not captured output.

Use It

code/main.py simulates a W2SG fine-tune on a synthetic 3-feature classification task. The gold label is a linear rule over all three features. The weak labeler thresholds on x[0] alone and then flips its label with probability 1 - weak_accuracy, so it never sees the other two signals. The strong model is a linear classifier fit by SGD. For each of weak_accuracy = 0.60, 0.70, 0.80, 0.90, the script trains the strong model on gold labels (the ceiling) and on weak labels, then prints weak-alone accuracy, both strong accuracies, and PGR.

Check: Use It0/4

1. In the simulator, the weak labeler thresholds x[0] and then flips its label with probability 1 - weak_accuracy. With weak_accuracy = 0.60, how often is the thresholded label flipped?Apply

Two steps from the weak labeler: first threshold x[0] alone, ignoring the other two features, then keep that label with probability weak_accuracy or flip it otherwise, with a dashed box asking the flip rate and whether it depends on the other two features. — Mechanism as given in main.py's weak_label function; rate not computed here.

2. For each weak_accuracy value, what does the simulator train and print?Apply

Train data splits into two branches, one labeled gold and one labeled by weak_label, each feeding its own call to train_strong, with both arrows converging on a dashed box asking what gets printed. — Typed schematic, not captured output.

3. You run the script once and the PGR values do not rise smoothly with weak_accuracy. What should you do before explaining the shape of the curve?Analyze

A bar chart of PGR against weak_accuracy 0.6, 0.7, 0.8 and 0.9 from one real run of main.py, where the bars go up, down, up, then down instead of climbing steadily. — Captured from python main.py (this repo's code, seed 29). One seed; weak-alone accuracy is measured with fresh random flips each time.

4. In the Use It simulator, why is a positive PGR weak evidence for Burns et al.'s claim about real models?Analyze

Two boxes, a weak labeler described as thresholding x[0] then applying a random flip, and a strong model described as linear and starting at weights of zero, with a dashed box asking whether a positive PGR here is strong evidence for Burns et al.'s claim about real models. — Both facts as stated in main.py and the Author's Note.

Ship It

This lesson produces outputs/skill-w2sg-pgr.md. Given an oversight setup description, it identifies the weak supervisor, the strong model, and the supervision quality, then computes PGR or asks for the numbers it needs. It also flags whether the claim is "weak can supervise strong" or "weak + oversight mechanism can supervise strong."

Check: Ship It0/1

1. A startup claims its setup 'lets a weak supervisor align a stronger model.' What should you establish before accepting the claim?Design

A quoted claim that a setup lets a weak supervisor align a stronger model, with a dashed box below asking what to establish before accepting it. — Claim quoted exactly as given in the stem.

Author's Note

I'd read the weak labeler in main.py before running anything. weak_label looks only at x[0] and then flips the result at random, so its mistakes are independent noise on top of a rule that ignores the other two features. A linear model trained on those labels can average the flips out, and I read that as a large part of what the PGR column is measuring here. It is a different thing from learning the intended task.

I'd also hold the numbers loosely. The script uses one seed and draws fresh weak noise when it measures weak-alone accuracy, so the PGR values I got did not rise smoothly with weak_accuracy. Exercise 1 asks for a curve, and I'd rerun with several seeds before explaining its shape.

The closing printout credits the strong model's pre-trained priors. The strong model in this file is a three-weight linear classifier that starts at zero, so there are no priors in play. That part of Burns et al. needs a real pretrained model, which this toy cannot show.

Exercises

Run code/main.py. Report PGR for weak_accuracy = 0.60, 0.70, 0.80. Explain the shape of the PGR curve.
Modify the weak labeler to have structured error (e.g., always wrong on a specific input class). Does PGR increase, decrease, or stay the same? Explain.
Read Burns et al. 2023 Section 4.3 (NLP tasks). Reproduce the "confidence auxiliary loss" intuition: when the strong model is more confident than the weak labels, who wins?
Design a scalable-oversight protocol that combines debate and task decomposition for a software-engineering task. Name one failure mode of each component and explain how the combination addresses or fails to address each.
Articulate what would falsify the "weak-to-strong generalization is a viable path to superalignment" claim. Be specific about the empirical signature you would need to see.

Key Terms

Term	What people say	What it actually means
Scalable oversight	"making the overseer stronger"	Mechanisms that increase an overseer's ability to evaluate a more-capable model
W2SG	"weak supervises strong"	Fine-tuning a strong model on weak labels and measuring the capability recovered
PGR	"performance gap recovered"	(fine-tuned - weak) / (ceiling - weak); 1.0 = fully closed, 0 = no help
Debate	"two U instances argue"	Scalable oversight mechanism where a weak judge picks between two U defenders
RRM	"recursive reward modeling"	U helps train the reward model for U+1; overseer capability tracks U
Task decomposition	"sub-tasks the human checks"	Break a hard task into sub-tasks the human can verify, recursively
Superalignment	"aligning superhuman AI"	The research agenda concerned with aligning models the human cannot directly evaluate