--- title: "RLHF and DPO: how models learn what people prefer" description: "Label response pairs and watch two preference-optimization pipelines, RLHF with a reward model and DPO's direct update, reshape a conceptual policy landscape." published: 2026-07-18 updated: 2026-07-18 canonical: https://protrailblazer.com/posts/rlhf-vs-dpo/ tags: [machine-learning, demo, rlhf, dpo, preference-optimization, reward-model, llm-alignment, kl-divergence] --- # RLHF and DPO: how models learn what people prefer **Answer:** RLHF and DPO are the two main recipes for teaching an LLM which of two answers people prefer. RLHF first fits a separate reward model to the preference labels, then updates the policy to chase that reward while a KL leash holds it near the original model. DPO skips the reward model and turns each labeled pair into a direct policy update. The demo below lets you label pairs yourself and replay the exact same labels through both pipelines. The interactive below is a preference-optimization training studio. Pick a prompt, choose a labeler persona, then tap the response you prefer. Each labeled pair flows into either the RLHF pipeline or the DPO pipeline and deforms a conceptual policy landscape in real time. Use the mode toggle to replay your exact labels through the other pipeline and see how the two approaches diverge from the same data. *Interactive demo: [open the rlhf dpo demo](https://protrailblazer.com/demos/rlhf-dpo/)* ## What you're seeing The screen is divided into three panels. On the left, the pipeline diagram shows the active training stages lighting up in sequence as you label: in RLHF mode the reward model fitting step activates first, then the policy update stage; in DPO mode only the direct policy update fires. A math panel behind the "show math" button displays the simplified objective for whichever mode is active. The center panel is the response studio. Four prompt presets (explanation, refusal, writing edit, factual) and four labeler personas (helpful, concise, formal, cautious) let you explore different preference shapes. Tap response A or B to label a pair, or let the auto-labeler run. The undo button removes the last pair; reset clears the dataset entirely. The right panel is the conceptual policy landscape, a 2-axis contour map drawn on canvas (concise to verbose on one axis, cautious to assertive on the other). Six response archetypes sit as fixed points in that space. The white ring is the frozen reference policy; the glowing dot is the current policy. As training steps accumulate, the dot drifts toward the style the persona prefers. The HUD at the top tracks labeled pairs, label agreement with the persona, the policy's probability on the preferred style, KL divergence from the base policy, total training steps, and the current loss. A few things to try: - **Switch the mode toggle from RLHF to DPO after labeling ten pairs.** The landscape replays your labels through the other pipeline from scratch. Any difference in where the policy dot ends up comes from the training machinery, not the data. - **Turn on label noise, then crank training strength to 2.0 or higher.** Watch the KL readout climb and the landscape warn that the policy is collapsing toward a single archetype. The KL leash in RLHF mode limits exactly this kind of overshoot. - **Swap personas mid-run.** Both pipelines drag the policy toward the new persona's preferred style immediately. The AGREEMENT stat measures consistency with the current persona's taste, not with any ground-truth notion of answer quality. - **Set batch size to 16 and run the auto-labeler.** The policy landscape moves in larger jumps: the batch accumulates 16 pairs before each update, so you see fewer, coarser steps rather than the fine-grained drift of batch size 1. - **Watch the RM LOSS versus DPO LOSS readout in the HUD.** In RLHF mode the loss is the reward model's cross-entropy on the labeled pairs; in DPO mode it is the DPO contrastive objective directly on the policy. They measure different things, so comparing raw numbers across modes is not meaningful. ## How RLHF and DPO actually work Before either method runs, the base model has already been through pretraining on a large text corpus. That gives it broad language ability. Preference optimization is a second stage that shapes the model's style and safety behavior without retraining from scratch. RLHF works in two steps. First, human annotators (or, in many current pipelines, a combination of human and AI raters) label response pairs: given prompt P, is response A or B better? Those labels train a separate reward model, typically a smaller transformer with a scalar head, to predict the preferred response. Then the policy model is updated using the reward model's scores as the training signal. A KL penalty keeps the updated policy from drifting too far from the original, preventing the model from gaming a reward that is itself imperfect. DPO, introduced by Rafailov et al. in 2023, collapses the two RLHF steps into one. It derives a closed-form reparameterization showing that the optimal policy for any reward can be written directly in terms of the policy's own log-probabilities and the reference policy. Training then uses a loss function that directly increases the relative probability of the preferred response versus the rejected one, with no reward model in the middle. The KL constraint is implicit in the formulation rather than a separate penalty term. Because DPO updates the policy directly on each pair, it only moves probability mass between the two responses in that pair. RLHF's reward model, by contrast, is a smooth generalizer: it fits a score surface over the entire style space and can push probability toward styles that never appeared in the labeled pairs, as long as the reward model extrapolates correctly. The demo makes this visible, but note that both the reward model and the policy landscape are conceptual simulations, not real model parameters. ## Why the KL leash matters Both RLHF and DPO anchor the updated policy to a frozen copy of the starting model called the reference policy. In RLHF the anchor is explicit: the training objective adds a KL penalty proportional to how far the new policy has moved from the reference. In DPO the same constraint is baked into the loss derivation. Without it, the optimizer would find reward-maximizing or pair-winning answers that look nothing like coherent language output. The KL readout in the HUD measures KL divergence in nats. When it rises sharply, the policy is collapsing onto one region of the style space, a sign that training strength is too high or the dataset is too noisy. Real implementations clip or anneal the training signal at this point. In the demo, turning on label noise and pushing hyperparameters like training strength high enough will trigger a collapse warning on the landscape panel. Understanding how these updates move the policy is connected to how the base model generates text in the first place. The [next-token generation walkthrough](https://protrailblazer.com/posts/how-llms-generate-text) covers how logits and softmax combine to produce a probability distribution over tokens, which is the exact distribution preference training reshapes. And for the optimization mechanics underlying both RLHF and DPO, the [gradient descent demo](https://protrailblazer.com/posts/gradient-descent-demo) shows how gradient descent drives weight updates toward lower loss. ## Frequently asked questions ### What is the difference between RLHF and DPO? RLHF trains a separate reward model on preference labels and then uses that model's scores to update the policy under a KL penalty. DPO skips the reward model and turns each labeled pair directly into a policy update using a contrastive objective. Both can train on identical preference data and produce similar results, but RLHF requires the extra reward model training step while DPO is a single-stage fine-tune. ### Why do RLHF and DPO produce different landscapes in the demo even with the same labels? The reward model in RLHF generalizes across the style space, fitting a smooth score surface that can influence styles beyond those in the labeled pairs. DPO's update is more local, directly adjusting the relative probability of the two responses in each pair. The conceptual landscape in the demo shows this divergence: the RLHF dot can drift toward styles that only the reward model extrapolates to, while the DPO dot tracks the labeled pairs more directly. Both landscapes are simulated and illustrative. ### What is the KL leash and why does preference training need one? KL divergence measures how far the updated policy has moved from the reference (starting) policy. Left unconstrained, a reward-maximizing optimizer or a contrastive loss will push the model toward extreme responses that score high on the training objective but have collapsed diversity or incoherent language. The KL penalty (in RLHF) or the implicit KL term in the DPO loss caps that drift at a tunable level. ### What does the labeler persona change in the simulation? Each persona has a preferred response archetype built into the auto-labeler. When you switch from "helpful" to "concise", the auto-labeler starts picking shorter, denser responses as the winner in each pair. Both pipelines then steer the policy toward that archetype. The AGREEMENT stat in the HUD counts how many of the labeled pairs agreed with the active persona's taste. Changing persona mid-run does not reset the dataset, so you can see two training regimes tug the policy in different directions. ### Is anything in this demo running a real model? No. All values, including the policy landscape, the reward model loss, the KL divergence, and the response text, are conceptual simulations running in the browser with no model weights. The sim is designed to make the algorithmic logic visible, not to reproduce empirically accurate numbers. Every stat is labeled as conceptual or simulated within the demo. ## Key takeaways - RLHF and DPO can train on the exact same preference pairs. The mode toggle replays your labels through the other pipeline, so any difference you see in the landscape comes from the training machinery, not the data. - A reward model is a generalizer. Because it fits one smooth score over the whole style space, RLHF can shift probability toward styles that never won a single comparison, while DPO's update only moves the two responses in each labeled pair. - More preference training is not automatically better. Push the training strength up and the KL readout climbs until the landscape warns that the policy is collapsing onto one style. The KL leash and the frozen reference exist to stop exactly that. - Preference data encodes the labeler, not the truth. Swap personas mid-run and both pipelines drag the same policy toward the new target, and the AGREEMENT stat measures consistency with the persona's taste, not answer quality.