LearnPython

The Direct Preference Optimization Family

Rafailov et al. (2023) showed that RLHF's optimum has a closed form in terms of the preference data, so you can skip the explicit reward model and optimize the policy directly. The idea spawned a family (IPO, KTO, SimPO, ORPO, BPO), each aimed at a failure mode of DPO. By 2026 direct alignment algorithms ship more frontier post-training runs than PPO. The over-optimization curve from Lesson 2 still applies, though: DAAs do not escape Goodhart, they just move where it bites.

TIME: ~75 minutesPREREQUISITES: Phase 18 · 01 (InstructGPT), Phase 18 · 02 (Reward hacking), Phase 10 · 08 (DPO basics)

Learning Objectives

Derive the DPO closed form from the RLHF-with-KL optimum.
State the failure mode each of IPO, KTO, SimPO, ORPO, BPO fixes in DPO.
Distinguish "implicit reward gap" from "preference strength" and explain why IPO's identity mapping matters.
Explain why Rafailov et al. (NeurIPS 2024) find that DAAs over-optimize despite having no explicit RM.

The Problem

The RLHF objective (Lesson 1):

max_pi E_{x,y~pi} [ r(x, y) ] - beta * KL(pi || pi_ref)

has a known optimum:

pi*(y|x) = (1/Z(x)) * pi_ref(y|x) * exp(r(x, y) / beta)

So the reward is implicitly defined by the ratio of the optimal policy to the reference:

r(x, y) = beta * log(pi*(y|x) / pi_ref(y|x)) + beta * log Z(x)

Substitute this into the Bradley-Terry preference likelihood and the partition function Z(x) cancels because it depends only on x. What remains is a loss in the policy parameters alone, with no reward model needed. That is DPO.

The wrinkle is that the derivation assumes three things: the optimum is reachable, the preference data is in-distribution, and the reference policy is the true mode anchor. None of them hold exactly, and each family member targets a different violated assumption.

Warm-up0/3

1. A model assigns probability 0.30 to a response, and a frozen copy of the original model assigns 0.20 to the same response. What is the sign of log(0.30 / 0.20)?Warm-up

Two cards: a model assigning probability 0.30 to a response, and a frozen copy of that model assigning 0.20 to the same response, above a dashed box asking for the sign of log(0.30 / 0.20).

2. In RLHF, the objective subtracts a KL penalty between the policy and a frozen reference model. What is that penalty mainly for?Warm-up

A Policy box being updated and a Reference box holding the frozen starting model, joined by a dashed line labelled KL penalty, above a dashed box asking what the tether is mainly for.

3. A policy is optimized hard against an imperfect learned reward score. What usually happens to the true quality of its outputs?Warm-up

A rising solid curve labelled learned score, and a short dashed curve from the same start labelled with a dashed box reading true quality, over an x-axis labelled optimization pressure.

Check: The Problem0/3

1. What is DPO?Define

2. Why can DPO skip training an explicit reward model?Explain

3. When the reward r(x, y) = beta * log(pi*/pi_ref) + beta * log Z(x) is put into the Bradley-Terry likelihood, why does Z(x) disappear?Explain

Two reward formulas for y_w and y_l, each ending in the same beta times log Z(x) term, above a dashed box asking what happens to those terms when one reward is subtracted from the other.

The Concept

DPO (Rafailov et al., 2023)

L_DPO = -log sigmoid(
  beta * log(pi(y_w | x) / pi_ref(y_w | x))
  - beta * log(pi(y_l | x) / pi_ref(y_l | x))
)

What can go wrong:

The implicit reward gap beta * (log(pi/pi_ref)_w - log(pi/pi_ref)_l) is unbounded. A tiny preference can produce an arbitrarily large gap.
The loss drives chosen and rejected log-probs in opposite directions. It can push the chosen absolute log-prob down as long as the rejected falls faster. This is the Degraded Chosen Response phenomenon.
Out-of-distribution preferences (a rare pair compared against another rare pair) produce arbitrary implicit rewards.

IPO (Azar et al., 2024)

Identity Preference Optimization replaces the log-sigmoid with an identity mapping on the preference probability. The loss becomes a squared-error on a bounded target:

L_IPO = (log(pi(y_w | x) / pi_ref(y_w | x)) - log(pi(y_l | x) / pi_ref(y_l | x)) - 1/(2 beta))^2

The margin is bounded by 1/(2 beta), so preference strength and implicit-reward gap stay proportional and nothing blows up.

KTO (Ethayarajh et al., 2024)

Kahneman-Tversky Optimization drops pairwise structure entirely. It takes a single labeled output with a binary "desirable" or "undesirable" signal and maps it to a prospect-theory utility:

v(x, y) = sigma(beta * log(pi(y|x) / pi_ref(y|x)) - z_ref)

Gains and losses get different weights (loss aversion). The payoff is that you can train on unpaired data, which is far more plentiful.

SimPO (Meng et al., 2024)

Simple Preference Optimization aligns the training signal with generation. It removes the reference policy and normalizes log-likelihood by length:

L_SimPO = -log sigmoid(
  (beta / |y_w|) * log pi(y_w | x)
  - (beta / |y_l|) * log pi(y_l | x)
  - gamma
)

A margin gamma stabilizes training. Length normalization takes away the incentive to exploit DPO's length bias, where a longer y_w gives a larger log-prob gap by construction.

ORPO (Hong et al., 2024)

Odds-Ratio Preference Optimization adds a preference term to the standard SFT negative log-likelihood:

L_ORPO = L_NLL(y_w) + lambda * L_OR
L_OR = -log sigmoid(log(odds(y_w) / odds(y_l)))

There is no reference policy because the SFT term is the regularizer. You train in a single stage from the base model to the aligned model, with no separate SFT checkpoint.

BPO (ICLR 2026 submission, OpenReview id=b97EwMUWu7)

BPO names the Degraded Chosen Responses problem: DPO preserves the ranking y_w > y_l, but the absolute log-prob of y_w can still drop. BPO adds a single-line correction that penalizes downward moves on the chosen response. It reports +10.1% accuracy over DPO on Llama-3.1-8B-Instruct for math reasoning.

The universal result: DAAs still over-optimize

Rafailov et al. "Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms" (NeurIPS 2024) trained policies with DPO, IPO, and SLiC on multiple datasets across KL budgets. The gold-reward-vs-KL curves have the same Gao et al. peak-and-collapse shape. The implicit reward gets queried on out-of-distribution samples during training, and KL regularization does not stabilize that.

So DAAs do not escape Goodhart. They change the surface where it bites, from "reward model over-optimized" to "reference policy ratio over-optimized." The fixes are the same for both: better data, ensembles, early stopping.

Choosing among them (2026)

If you have large paired preference data: DPO with conservative beta, SimPO if length bias is evident.
If you have unpaired binary feedback: KTO.
If you want a single-stage pipeline from a base model: ORPO.
If you see degraded chosen log-probs in DPO logs: BPO.
If preference strengths vary widely and DPO is saturating: IPO.

Run all five on a battery of your own tasks and pick the winner per task. Math reasoning and safety have no reason to share the same optimum.

Check: The Concept0/25

1. In DPO, what is the implicit reward of a response y?Define

A Policy box with a real probability value and a Reference box with a different real probability value for the same action, above a dashed box asking for the implicit reward of this response.

2. What kind of training data does KTO use, and how does it score it?Define

Two separate, unpaired examples, one action labelled undesirable and a different action labelled desirable, with no pairing between them, above a dashed box asking how they are scored.

3. What does the term direct alignment algorithm (DAA) cover?Define

Six name chips, DPO, IPO, KTO, SimPO, ORPO and BPO, above a dashed box asking what all six skip in common.

4. In IPO, the implicit reward gap between chosen and rejected responses is capped at what value?Define

Two small charts of a gap over training steps. The left, labelled DPO, keeps climbing with no ceiling. The right, labelled IPO, levels off at a dashed line marked with a question mark. — Shape only; no value is drawn for the level IPO settles at.

5. Which property is unique to SimPO compared to DPO?Define

A two-row table for DPO showing it uses a reference policy and raw log-likelihood, both yes, above a two-row table for SimPO with the same two properties both marked with a question mark.

6. What does ORPO (Hong et al., 2024) do differently from DPO in terms of training stages?Define

A typical pipeline of four boxes, base model, SFT checkpoint, DPO, aligned model, above ORPO's pipeline: base model, a dashed box asking how many stages and whether there is a checkpoint, and aligned model.

7. What does BPO change relative to DPO?Define

8. What is the 'Degraded Chosen Response' problem in DPO that BPO targets?Define

A chosen-response log-prob box and a rejected-response log-prob box, both starting equal to the reference, each with its own dashed box asking what happens after one DPO step.

9. Besides the unbounded implicit reward gap and Degraded Chosen Response, what third DPO failure mode is listed under The Concept (DPO)?Define

A three-row list of DPO failure modes: unbounded implicit reward gap and Degraded Chosen Response marked as given, and a third row with a question mark, highlighted.

10. What failure mode does IPO (Azar et al., 2024) primarily address in DPO?Explain

11. Why does SimPO's division of log-likelihood by response length remove an incentive that DPO has?Explain

A short y_w of 3 tokens and a long y_w of 9 tokens, above a dashed box asking how raw, un-normalized log-likelihood scales with length.

12. ORPO has no reference policy. What keeps the policy from drifting without one?Explain

Two boxes, L_NLL(y_w) labelled standard SFT term, and L_OR labelled odds-ratio preference term, above a dashed box asking which one keeps the policy from drifting without a reference policy.

13. A pair with only a slight preference can give DPO an arbitrarily large implicit reward gap. What does IPO change so the gap tracks preference strength?Explain

A chart of DPO's implicit reward gap climbing steadily over training steps with no ceiling, and a separate dashed horizontal line labelled target with a question mark.

14. Which statement correctly separates preference strength from the implicit reward gap?Explain

Quantity 1 described as a property of the preference data, how decisively it favors y_w, and Quantity 2 described as a property of the policy, built from beta and log-probability ratios, above a dashed box asking which is preference strength and which is the implicit reward gap.

15. Rafailov et al. (NeurIPS 2024) on scaling laws for direct alignment algorithms show:Explain

A gold-reward-versus-KL-budget axis with three dashed boxes side by side, labelled DPO, IPO and SLiC, each with a question mark for its curve's shape.

16. Why does KL regularization fail to stop direct alignment algorithms from over-optimizing?Explain

A Policy box tethered to a Reference box by a dashed KL line, with a separate dashed line pointing up to a box reading implicit reward query, far from the training data, above a dashed box asking whether the KL tether reaches and stabilizes that query.

17. Using the IPO loss, suppose beta = 0.5 and the chosen log-ratio minus the rejected log-ratio equals 0.4. What is the loss?Apply

Given beta = 0.5 and a chosen-minus-rejected log-ratio gap of 0.4, a dashed box asking for the IPO loss. — The two given numbers come from the question.

18. The log-ratio gap is already well above 1/(2 beta). How do IPO's and DPO's gradients on that pair differ?Analyze

A number line with a target tick and a gap marker already placed to the right of the target, with two dashed boxes above asking what IPO's next step and DPO's next step do to it.

19. Your DPO run on paired math data improves held-out pairwise accuracy, but logs show the chosen response's log-prob dropping below its reference value on most pairs. Which change targets this directly?Analyze

Arrow directions are illustrative of 'most pairs'; no specific log-prob numbers are drawn.

20. After DPO on paired data, your model's answers got much longer, and the chosen responses in the data are longer on average. What is the likely cause and fix?Analyze

A box for chosen training responses, longer on average, and a box for rejected training responses, shorter on average, with a note that the model's own answers got much longer after training, above a dashed box asking for the likely cause and fix.

21. A teammate moves from DPO to IPO and says over-optimization is now solved because neither method trains a reward model. What is wrong with this?Analyze

A quoted claim that moving from DPO to IPO solves over-optimization since neither trains a reward model, with two boxes confirming both DPO and IPO train no reward model, above a dashed box asking whether the claim's conclusion is right.

22. You hold a base model with no SFT checkpoint plus paired preference data, and you want a single training run. Which loss fits and why?Design

Three stacked requirement boxes: base model with no SFT checkpoint, paired preference data available, and wanting a single training run, above a dashed box asking which loss fits and why.

23. Your paired data mixes near-ties with lopsided wins, and DPO's loss is saturating. Which method does the lesson point to?Design

A histogram of preference strengths with a cluster near 0.5 and a cluster near 1.0, under a note that DPO's loss is saturating on this mixed data, above a dashed box asking which method the lesson points to.

24. You must align one model for math reasoning and for safety. How should you pick a preference loss?Design

A Math reasoning box and a Safety box, each asking for its own best preference loss, above a dashed box asking how the two choices should relate.

25. Your DPO run's gold reward peaks and then falls as KL from the reference grows. Which response matches the lesson?Design

A single curve for DPO's gold reward against KL from the reference, rising to a peak and then falling, above a dashed box asking for the best response. — Shape matches exactly what the question's own stem describes for DPO; no other method's curve is drawn.

Use It

code/main.py compares six losses (DPO, IPO, KTO, SimPO, ORPO, BPO) on a toy preference dataset where the true preference strength varies by pair. The five pairwise losses train on the same 500-pair sample and KTO on 500 unpaired labels, all with a small softmax policy. It prints each method's final probabilities, win rate, and logits.

Check: Use It0/7

1. In train_dpo the policy starts as a copy of the reference. On the very first step, what are `margin` and the DPO `g_margin`?Apply

The train_dpo margin and dpo-branch g_margin formulas, with pi starting as a copy of ref, above a dashed box asking for their values on the first step. — Formulas quoted from main.py's train_dpo.

2. In the IPO branch with the default beta = 0.1, what is `target`, and what is `diff` on the first step when the policy equals the reference?Apply

The train_dpo IPO-branch diff formula with beta = 0.1 and target left as a fixed value set from beta, above a dashed box asking for target and diff on the first step. — Formula quoted from main.py's train_dpo, ipo variant; the line that sets target from beta is masked here.

3. In train_kto, the model's `value` is 0. How do the gradient weights `g_value` compare for a desirable and an undesirable example?Apply

The train_kto formulas for value = 0, giving desirable and undesirable gradient weights g_value, above a dashed box asking how the two compare. — Formulas quoted from main.py's train_kto.

4. In train_simpo with beta = 1.5 and gamma = 0.5, log_pi_w = -1.0 and log_pi_l = -1.5 (lengths are 1). What is `margin`?Apply

The train_simpo margin formula with beta 1.5, gamma 0.5, log_pi_w -1.0 and log_pi_l -1.5, above a dashed box asking for margin. — The numbers come from the question.

5. In the bpo branch, the chosen log-ratio (log_pi_w - log_ref_w) is -0.4. What does the anchor term do to the chosen log-prob?Apply

The bpo branch formulas with the chosen log-ratio given as -0.4, above a dashed box asking the net effect on the chosen log-prob. — The log-ratio comes from the question.

6. In the toy run, action 0 is usually preferred over actions 2 and 3, yet DPO lowers its probability from 0.273 to 0.231, while BPO ends at 0.253. Which failure is this?Analyze

Three bars for action 0's probability under REF, DPO and BPO, each labelled with the exact value the question gives, above a dashed box asking which failure this is. — The three values are exactly the ones the question states.

7. In the toy run SimPO ends close to the other methods. Why can this run not show that length normalization works?Analyze

The real lens list from main.py's train_simpo, set to four ones, above a dashed box asking why this cannot demonstrate length normalization. — Line quoted verbatim from main.py.

Ship It

This lesson produces outputs/skill-preference-loss-selector.md. Given dataset statistics (paired vs unpaired, variable vs uniform preference strength, length distribution) and a target (single-stage or SFT-then-preference), recommend a preference loss and report the failure mode it protects against.

Author's Note

I'd run main.py before rereading the formulas. It is a 4-action softmax policy, not a language model, so the six losses sit side by side in a few lines each and you can read the gradient of every one. It prints the probabilities, win rate, and logits per method. Chosen-log-prob drift is not printed, so for Exercise 1 you'll compute it yourself from the logits.

The toy is also honest about its limits. There is one token per action, so SimPO's length normalization divides by 1 and cannot show the length-bias fix. The BPO penalty is a hand-picked 0.05 weight on the chosen log-ratio, and the ORPO term is a simplified version of the odds-ratio loss. I'd treat those three as shapes of the idea, and test the length claim with Exercise 3 before believing it.

My bet is that the part worth sitting with is the implicit reward, beta * (log pi - log pi_ref). DPO, IPO, and BPO all push on it in train_dpo, and what changes between them is the gradient on it: DPO saturates through the sigmoid, IPO regresses it toward 1/(2 beta), and BPO adds a pull on the chosen log-ratio. Once that is clear, the rest of the family reads as variations on where you put the anchor.

Exercises

Run code/main.py. From the printed logits, compute log pi(a) - log pi_ref(a) for every action under DPO and BPO. Action 0 is usually preferred over actions 2 and 3, yet DPO lowers its probability (0.273 to 0.231). Check whether BPO holds it higher (0.253). Then check the best action, 1, where BPO ends below DPO (0.507 vs 0.636), and explain why the 0.05 anchor weight does that.
Modify the preference data so that all pairs have equal strength. Which of the six methods is most robust? Which degrades? Explain IPO's advantage here.
Make the rejected responses on average 2x longer than chosen. Without changing anything else, show DPO's length exploitation numerically and SimPO's fix.
Rafailov et al. (NeurIPS 2024) claim DAAs over-optimize. Reproduce a single-point version: plot chosen-minus-rejected KL divergence and observe over-optimization in DPO at large beta.
Read the BPO paper abstract (OpenReview b97EwMUWu7). Write down the one-line correction BPO adds to DPO. Confirm against the implementation in code/main.py.

Key Terms

Term	What people say	What it actually means
DPO	"RLHF without a reward model"	Loss derived from the closed-form RLHF optimum; policy parameters only
Implicit reward	"the log-ratio"	`beta * log(pi(y\|x) / pi_ref(y\|x))`, the DPO-implied reward
IPO	"bounded DPO"	Replaces log-sigmoid with identity; implicit reward gap capped by `1/(2 beta)`
KTO	"unpaired DPO"	Prospect-theory utility over single labels with loss aversion
SimPO	"reference-free DPO"	Length-normalized log-likelihood + margin; no reference policy
ORPO	"one-stage DPO"	NLL + odds-ratio preference term; trains from base model in one pass
BPO	"chosen-preserving DPO"	DPO plus a penalty for decreasing the chosen response's absolute log-prob
Degraded Chosen	"chosen goes down"	DPO decreases chosen log-prob so long as rejected falls faster
DAA	"direct alignment algorithm"	Any preference-loss method that skips an explicit RM