Instruction-Following as Alignment Signal

Pre-trained language models complete text. They do not answer questions. Ask GPT-3 "write a Python function that reverses a list" and you often get back another prompt, because most of the training distribution is web text that continues with more web text. The model is doing its job. The job is wrong.

The fix the labs settled on is a proxy: human preference. Two completions go to a rater, the rater picks the better one, and a reward model learns the rater. Then an RL loop shifts the policy toward outputs the reward model scores high. That is the full InstructGPT thesis in three sentences. The rest of the paper is engineering.

Stage 1: supervised fine-tuning (SFT)

Collect prompt-response pairs where the response is what a well-intentioned human would write. Ouyang et al. used 13k prompts from labelers and the OpenAI API. Fine-tune the base model on this data with standard cross-entropy loss.

What SFT gives you: the model now answers questions instead of continuing them. What it does not give you: any signal about which answer the rater prefers when multiple are plausible.

Stage 2: reward model (RM)

For each prompt, sample K completions from the SFT model. A labeler ranks them. Train a reward model that scores any prompt-response pair so that, for pairs where y_w was preferred over y_l:

L_RM = -log sigmoid(r(x, y_w) - r(x, y_l))

This is the Bradley-Terry pairwise preference loss. The RM is usually initialized from the SFT model with the LM head replaced by a scalar head.

Reward models are small: 6B was enough for the 175B InstructGPT. They are also fragile. Section 5 of the paper is mostly about reward-hacking behaviours that showed up at small scale.

Stage 3: PPO with a KL penalty

Define the objective:

J(pi) = E_{x~D, y~pi(.|x)} [ r(x, y) ] - beta * KL(pi(.|x) || pi_SFT(.|x))

Maximize with PPO. The KL term keeps pi from drifting far from the SFT policy. Without it, the optimizer finds adversarial examples: strings that score high under the RM because the RM never saw them, not because humans actually prefer them.

The KL coefficient beta is the single most important RLHF hyperparameter. Too low: reward hacking. Too high: no improvement over SFT.

The alignment tax

After RLHF, the model is preferred by humans but regresses on standard benchmarks (SQuAD, HellaSwag, DROP). Ouyang et al. call this the alignment tax and fix it with PPO-ptx: mix pre-training gradients into the RL objective so the model does not forget how to do downstream tasks it was never rewarded for.

J_ptx(pi) = J(pi) + gamma * E_{x~D_pretrain} [ log pi(x) ]

PPO-ptx became standard. Anthropic, DeepMind, and Meta all use some variant.

The result

A 1.3B InstructGPT (SFT + RM + PPO-ptx) is preferred by labelers over the 175B base GPT-3 about 70% of the time. The gap widens on hidden-test prompts from production traffic. Two things to read off this number:

1. Alignment is a different axis from capability. The 175B model had more capability; the 1.3B model had more alignment; labelers preferred the aligned one.
2. The capability floor is set by the base model. You cannot RLHF a base model into knowing facts it never saw.

Why this is the reference point for Phase 18

The critiques in later lessons (reward hacking in Lesson 2, DPO in Lesson 3, sycophancy in Lesson 4, CAI in Lesson 5, sleeper agents in Lesson 7, alignment faking in Lesson 9) each argue against some part of this pipeline. Reward hacking attacks stage 2. DPO collapses stages 2 and 3. CAI replaces the human labeler. Sycophancy shows the labeler is a biased signal. Alignment faking shows the policy can route around stage 3 entirely. None of them make sense until the pipeline is in your head.

main.py simulates the three stages on toy preference data. The base "policy" is a softmax over three actions {A, B, C}. Stage 1 SFT fits the action frequencies of 200 labeler demonstrations in closed form. Stage 2 fits a Bradley-Terry reward model from 500 pairwise rankings. Stage 3 runs a stripped-down policy-gradient update (REINFORCE with a KL-shaped reward, standing in for PPO) with a KL penalty to the SFT policy. You can watch the reward climb, the KL divergence grow, and the policy drift. Then run it with the KL term off and compare.

With the fixed seed, beta = 0.1 and beta = 0.0 end up close together, both putting roughly 97% of the probability on B. The reward model is accurate on three actions, so there is little for the optimizer to exploit. The KL term earns its keep when the reward model is wrong.

What to look at:

• Reward trajectory with beta = 0.1 vs beta = 0.0.
• KL(pi || pi_SFT) over training steps.
• Final action distribution compared to labeler preference.

This lesson produces outputs/skill-instructgpt-explainer.md. Given an RLHF pipeline description or a paper abstract, it identifies which of the three stages is being modified, what loss is being used at each stage, and whether a KL penalty or equivalent regularizer is present.

1. Run main.py. Set beta = 0.0 and report the action distribution after the default 300 policy-gradient steps. Explain the mode-seeking behaviour in one paragraph.

2. Modify the reward model to have a +0.5 bias for action A (a simulated reward bug, matching the buggy run in main.py). Run the stage 3 loop with beta = 0.1. Does the KL penalty prevent the policy from exploiting the bias? At what beta does exploitation become visible?

3. Read Ouyang et al. (arXiv:2203.02155) Figure 1. Reproduce the labeler-preference curve by running PPO for 1, 5, 20, 100 steps and measuring preference against the SFT model.

4. The paper's Section 4.3 reports a 1.3B InstructGPT beats 175B GPT-3 about 70% of the time. Why would the ratio be higher on hidden production prompts than on the labeler's own prompts?

5. Replace the PPO loss with DPO (Phase 10 · 08) on the same preference data. Compare final policy drift (KL to SFT) and final reward. Which method drifts further at matched reward?

• Ouyang et al. Training language models to follow instructions with human feedback (arXiv:2203.02155): the InstructGPT paper, foundation for the RLHF pipelines that followed
• Stiennon et al. Learning to summarize from human feedback (arXiv:2009.01325): the RLHF-for-summarization predecessor
• Christiano et al. Deep reinforcement learning from human preferences (arXiv:1706.03741): the original preference-based RL formulation
• Bai et al. Training a Helpful and Harmless Assistant with RLHF (arXiv:2204.05862): Anthropic's HH extension of the InstructGPT pipeline