RLHF vs DPO: How Models Learn Human Preferences

Supervised fine-tuning teaches a model to imitate good answers. It cannot teach the model that one answer is better than another — and that gap is exactly where sycophancy, hallucination, and bland style come from. Show a model a thousand polished responses and it learns to copy their surface; it never learns the boundary between a response you'd ship and one you'd reject.

RLHF and DPO both close that gap by training on pairs: a chosen response and a rejected one for the same prompt. RLHF does it the way ChatGPT was built — a reward model plus reinforcement learning, four models resident in memory, unstable and expensive. DPO does it with a single contrastive loss on a fixed dataset, two models, no reward model at all, and lands within roughly 90–95% of RLHF's quality.

This walks the mechanics behind both: why imitation can't express preference, the InstructGPT recipe and the β KL leash that keeps it honest, the DPO loss decoded number by number, and the one knob people get wrong — a learning rate that must be 10–100× smaller than SFT.

Why SFT can't say "B is better than A"#

SFT minimizes negative log-likelihood on a target response. For a chosen sequence, the loss is L_sft = −Σ_t logπ_θ(token_t) — sum the per-token log-probabilities, negate, push them up. The gradient touches only the tokens in the target. There is no term for a rejected response, so its gradient is literally zero:

Preference learning changes the shape of the signal. Instead of one response to copy, you feed the model a pair and ask it to widen the gap: make the chosen response more likely relative to the rejected one. Now the rejected column has a nonzero gradient — pushed down — computed from the same log-probabilities SFT ignored. That single change, from per-token imitation to per-pair contrast, is the whole idea both RLHF and DPO implement.

The RLHF pipeline: SFT → reward model → PPO#

RLHF is the InstructGPT recipe, and it runs in three stages that must happen in order.

Stage 1 — SFT. Start from an instruction-following model, never a raw base model. RLHF needs reasonable responses to compare; Base → SFT → RLHF is mandatory, Base → RLHF doesn't work.

Stage 2 — Reward model. Collect preference pairs (prompt, chosen, rejected). Take the SFT model, drop its language-model head, bolt on a single linear layer that outputs one number, and train it with the Bradley-Terry objective so score(chosen) > score(rejected). That number is a learned proxy for human judgment.

Stage 3 — PPO loop. The policy generates responses, the reward model scores them, and PPO nudges the policy toward high-scoring outputs. New responses are generated every step — this is online training. The core reward at each step is:

Count the models this needs in memory: the policy being trained, a frozen reference (the SFT snapshot) to measure KL against, the reward model, and a value/critic network PPO uses to estimate advantages. Four models resident at once. That's the memory bill people quote when they say RLHF is expensive.

The β leash: why the KL term exists#

The reward model is only a proxy. Optimize any proxy hard enough and it stops tracking the thing you cared about — Goodhart's Law. Without a constraint, PPO finds responses that score high on the reward model while being genuinely worse, drifting far from anything the model was trained to produce. That's reward hacking.

The −β·KL term is the entire defense. Every candidate response is scored by a constrained reward J_i = r_i − β·KL_i, and the policy effectively takes the argmax. KL measures how far a response has drifted from the frozen reference. Once drifting costs more than the fake reward pays, the policy stops chasing the proxy.

Watch it work on four candidates — anchor sits at the reference, genuine is genuinely good, hacker is the Goodhart trap (high proxy reward, low true quality, big drift), verbose is a middling reward-chaser:

At β=0 there's no leash: pure reward maximization picks the hacker (J=0.95), and true quality collapses to 0.20. At β=0.5 the KL cost of the hacker's 1.2 drift finally outweighs its fake reward, and the selection flips to genuine (true quality 0.90) — this is the sweet spot. At β=1.0 the leash is so tight that every move costs more KL than it earns, and the policy is stuck at the anchor. β too low is reward hacking; β too high is a model that can't improve. Tuning it is the central RLHF craft.

DPO: skip the reward model entirely#

Direct Preference Optimization asks a sharp question: if the reward model is just a function of log-probabilities anyway, why train a separate network and run PPO? DPO derives a loss that optimizes the same preference objective directly on the policy, using only two models — the policy and a frozen reference. No reward model, no critic, no online generation.

The trick is an implicit reward. The reward of a response is how much more likely the policy makes it than the frozen reference — a log-space subtraction, not a linear ratio:

That − logπ_ref term is not decoration. Subtracting the reference is what pins the policy near it: the absolute log-probabilities cancel, and only the relative movement survives. The reference term IS the KL constraint — the same leash RLHF spends a whole online loop enforcing is baked into DPO's loss for free.

Walkthrough: one DPO update, number by number#

Take one preference pair with real per-response log-probabilities and β = 0.1. The chosen response: policy logprob −1.2, reference −2.0. The rejected: policy −3.0, reference −2.5.

Read it top to bottom. The policy already likes the chosen response more than the reference does (Δ̂ = +0.80) and the rejected one less (Δ̂ = −0.50), so the raw reward gap is 1.30. β = 0.1 scales that into a margin of +0.13. Positive margin means the pair is already ordered correctly, and σ(0.13) = 0.532 — the model assigns a 53% probability to "chosen ≻ rejected", just above a coin flip. The loss −log σ is 0.630, a hair below log 2 ≈ 0.693 (the tie point), confirming it's marginally better than chance.

The last row is the payoff. Every parameter update is scaled by g = σ(−margin) = 0.468. When a pair is already ranked correctly with a large positive margin, g → 0 and the example stops contributing — it's effectively learned. When a pair is inverted (negative margin), g → 1 and the update is maximal. DPO builds its own curriculum: hard-and-wrong examples dominate the gradient, easy-and-right ones fade out, with no reward model or KL scheduler to babysit.

RLHF vs DPO: the honest comparison#

Online generation is RLHF's real advantage: the policy explores responses the fixed dataset never contained, which is why the very best models still lean on it. DPO trades that exploration for simplicity and stability, and for most fine-tuning it's the right trade — you get the large majority of the alignment gain from a loss you can debug like any classifier.

Why DPO's learning rate must be 10–100× lower#

This is the mistake that quietly ruins DPO runs. The contrastive −log σ(β·θ) surface is far steeper near its minimum than the SFT NLL surface. Gradient descent is stable only while the step size stays under η_crit = 2 / curvature at the minimum; past that bound the iterates oscillate and then diverge.

An SFT-scale learning rate (around 2e-4) sits comfortably inside the SFT basin but lands past the stability bound on the steeper DPO surface — the loss climbs instead of settling. A DPO-scale rate (around 5e-6, roughly 40× smaller) stays under η_crit and converges. It's not superstition or a safety margin; it's the stability threshold of a sharper loss. Start DPO around 5e-6 to 1e-5 and only move up if the loss is descending too slowly.

What to remember#

• SFT imitates one response; preference learning contrasts a pair. Only the second can express "B is better than A", because only it puts a nonzero gradient on the rejected sample.
• RLHF = SFT → reward model → PPO, four models in memory, online, R = RM_score − β·KL. The KL term is the sole defense against reward hacking; β too low hacks, β too high freezes.
• DPO drops the reward model and PPO for a single loss, −log σ(β·(Δ̂_chosen − Δ̂_rejected)), with the frozen-reference term standing in for the KL leash.
• DPO's per-example gradient weight g = σ(−margin) self-regulates: maximal when wrong, ~0 when aligned.
• Use a DPO learning rate 10–100× below SFT — the contrastive surface's stability bound demands it.

Keep going#

This is Part 3 of the fine-tuning track. Open the interactive deep dive to step the DPO loss live, tune β and watch reward hacking flip to alignment, and drag the learning rate until real SGD diverges past the stability bound — explore it interactively on YeetCode.

Related reading:

• RLVR: Reinforcement Learning with Verifiable Rewards, Explained — what happens when the reward is a checker, not a learned model.
• RL Environments for LLM Agents: Where Models Practice — the online-training substrate RLHF-style methods need.
• From APIs to Agents: Function Calling and the ReAct Loop — what a preference-tuned model does once it's aligned.
• Agent = Model + Harness: Context Engineering and Evals — measuring whether the alignment actually held.

FAQ#

What is the main difference between RLHF and DPO?#

The dividing line is whether preference gets its own network. RLHF externalizes the signal — a trained reward model plays judge, and a reinforcement-learning loop chases its scores while a KL penalty holds the leash. That setup demands a policy, a reference, a reward model, and a critic all resident at once, and the loop can collapse if any piece is mistuned. DPO keeps everything algebraic: it exploits the closed-form link between reward and optimal policy to fold the whole objective into one classification-style gradient step, so two models and a fixed dataset replace the judge, the critic, and the sampling loop. You trade a few points of top-end quality for something you can debug like an ordinary classifier.

Why does RLHF need a KL divergence penalty?#

A reward model is only trustworthy in the neighborhood where it actually saw preference data. Push the policy far from the SFT distribution and its scores become extrapolations — confident numbers with nothing behind them, which is precisely the region where reward hacking lives. The KL penalty acts as a trust region: it puts a price on every unit of distance from the reference so the optimizer stays where the proxy's judgment still means something. β sets that price. Set it too cheap and the loop buys its way into high-scoring nonsense; set it too expensive and no move is ever worth its cost, so the policy never leaves the starting point.

How does DPO enforce a KL constraint without a reward model?#

The constraint falls out of the derivation instead of being bolted on. RLHF's KL-regularized objective has a known closed-form optimum, and inverting that relationship lets you write the reward purely as a scaled log-ratio between policy and reference. Substitute that expression into the preference loss and the leash is already sitting inside it — nothing left to schedule or tune separately. Because the loss can only ever reward lifting the chosen response's ratio above the rejected one's, and never the raw probability, wandering away from the reference stops paying off. That is the same discipline the explicit penalty imposes in RLHF, obtained as a side effect of the algebra.

Why does DPO use a much smaller learning rate than SFT?#

The two losses have very different geometry. SFT's likelihood objective sits in a wide, forgiving basin, so a large step still tends to land somewhere lower. The log-sigmoid contrast in DPO carves a much narrower valley — a step sized for SFT's width overshoots the bottom and climbs the opposite wall, and the loss ratchets upward until it blows apart rather than settling. The fix isn't timidity for its own sake; it's matching the step to the valley's width. In practice that means starting one to two orders of magnitude lower, around 5e-6 to 1e-5, and only nudging up if convergence is genuinely sluggish.

Does DPO fully replace RLHF?#

For most teams, effectively yes — but not at the very top. DPO learns only from the pairs you gathered up front, so its ceiling is locked in the moment training begins. RLHF keeps sampling the policy's own current outputs, which lets it surface and correct failure modes that appear only after the model has already shifted — behaviors no static file could have anticipated. That self-generated exploration is what buys the last few points of quality at the frontier. If you aren't chasing those points, DPO's stability and low cost make it the pragmatic default; if you are, and you can afford the online machinery, RLHF still earns its place.