04 Training and Adaptation 3 min read 669 words
RLHF and DPO (Alignment)
Alignment is the process of ensuring an LLM's behavior matches human values and instructions. In 2025, the field has moved from traditional RLHF to more efficient and scalable methods like DPO and Online RL.
The Alignment Problem
Pretrained models are "knowledgeable but uncontrolled." They may:
- Generate harmful content (Safety).
- Fail to follow instructions (Instruction Following).
- Hallucinate wildly (Factuality).
Alignment creates "Reward Models" and "Policy Updates" to steer the model.
RLHF: The Foundation
Reinforcement Learning from Human Feedback (RLHF) involves three steps:
- SFT: Supervised Fine-Tuning.
- Reward Model (RM): Train a model on
(Prompt, Winning_Response, Losing_Response)to predict human scores. - PPO (Proximal Policy Optimization): Use the RM to provide a "reward signal" to the LLM via Reinforcement Learning.
2025 Nuance: Traditional RLHF is now considered too complex/unstable for most teams due to the overhead of training a separate Reward Model and the instability of PPO.
DPO: Direct Preference Optimization
DPO is the 2024-2025 industry standard. It eliminates the Reward Model.
How it Works:
DPO uses the LLM itself as the Reward Model by mathematically deriving the optimal policy directly from preference data.
- Goal: Maximize the probability of the "winning" response and minimize the "losing" response, relative to a fixed "reference model."
The 2025 Multi-Stage Alignment Pattern:
- Base SFT: 5k-10k high-quality samples.
- DPO Step 1: Alignment for instruction following.
- DPO Step 2: Alignment for safety and specific tone.
Online Alignment (Dec 2025 Standard)
The Problem with Offline DPO: It only learns from static data. If the model improves beyond that data, it hits a ceiling.
The Solution: Online DPO (or RLOO):
- The model generates 4-8 responses to a prompt.
- A Judge Model (e.g., GPT-5.2) or a Rule-based Reward (e.g., Code Execution) ranks them in real-time.
- The model updates its policy immediately based on this "Online" feedback.
Alignment for Reasoning Models (o1/DeepSeek-R1 style)
Aligning "Thinking" models requires a shift from Response Preference to Process Preference.
| Feature | Standard Alignment | Reasoning Alignment |
|---|---|---|
| Reward Target | The final answer | The Chain of Thought (CoT) |
| Reward Signal | Helpful/Safe | Correctness + Conciseness |
| Method | Human Ranking | Rule-based (e.g., "Did the code run?") |
Principal-level Nuance: "Verification-based RL" is the secret to 2025 frontier models. Instead of humans saying what is better, we use hard verifiable outcomes (Math answers, Code test cases) as the reward signal.
Interview Questions
Q: Why is DPO often preferred over RLHF/PPO in 2025?
Strong answer: DPO is preferred primarily due to its simplicity and stability. PPO requires maintaining four models in memory (Policy, Reference, Value, and Reward), which is extremely VRAM-intensive. Furthermore, PPO is notoriously sensitive to hyperparameters and often suffers from "reward hacking" or sudden collapse. DPO treats alignment as a simple classification problem on preference pairs, making it much more robust, easier to tune, and significantly cheaper to run.
Q: What is the risk of "Alignment Tax"?
Strong answer: The "Alignment Tax" refers to the decline in a model's raw capabilities (e.g., coding, creative writing, or logical reasoning) after it is aligned for safety or specific personas. Because the model is being forced to prioritize safety or adherence to a specific style, it may become "too cautious" or lose the nuance it learned during pretraining. Modern 2025 techniques like Steerable Alignment and DPO-with-KL-penalty aim to minimize this by ensuring the model's policy doesn't drift too far from the original pretrained distribution.
References
- Rafailov et al. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (2023)
- Schulman et al. "Proximal Policy Optimization Algorithms" (2017)
- OpenAI. "Learning to Reason with LLMs" (2024)
Next: Knowledge Distillation
Key takeaways
01
DPO removes the reward model
PPO keeps four models in memory — policy, reference, value and reward — while DPO derives the optimal policy from preference pairs, turning alignment into a classification problem.
02
Alignment ships in stages
The described pattern is SFT on 5,000-10,000 high-quality samples, one DPO pass for instruction following, then a second DPO pass for safety and tone.
03
Static preference data has a ceiling
Offline DPO stops helping once the model outruns its dataset; online DPO samples four to eight responses per prompt and has a judge or rule rank them live.
04
Reasoning models reward the process
Alignment shifts from response preference to process preference: the chain of thought is scored on correctness and conciseness using verifiable outcomes such as whether the code ran.
05
Alignment carries a capability tax
Safety and persona tuning can erode coding, creative writing and reasoning; KL penalties and steerable alignment exist to keep the policy near the pretrained distribution.