PIPELINE · RLHFSIMPLIFIED
01 · COLLECT
Preference data
02 · FIT
Reward model
03 · UPDATE
Policy update π
ANCHOR
Reference π₀ · frozen
RLHF (two stages)
L(φ) = −log σ( r̂(y) − r̂(yₗ) )
max E[r̂(y)] − τ·KL(π ∥ π₀)
DPO (one stage)
L(θ) = −log σ( β[Δ(y) − Δ(yₗ)] )
Δ(y) = log π(y) / π₀(y)
Conceptual modelProduction RLHF and DPO train billion-parameter models on large datasets with evaluation and safety review. Two style axes stand in for many preference dimensions, and many variants exist.
RESPONSE PAIRPAIR #0
PROMPT “Why is the sky blue?”
CHOOSE A RESPONSE, OR LET THE AUTO LABELER RUN
POLICY LANDSCAPECONCEPTUAL PROJECTION