Reinforcement Learning from Human Feedback (RLHF)

RLHF operates through a sequential, three-phase architecture:

• Stage 1 — Supervised Fine-Tuning (SFT): The base model is fine-tuned on high-quality demonstration data where human labelers write ideal responses to prompts. This establishes a baseline of desired behavior.
• Stage 2 — Reward Model (RM) Training: Human annotators rank multiple model outputs for the same prompt (e.g., A > B > C). A scalar reward model is trained to predict these preference rankings, effectively learning a proxy for human judgment.
• Stage 3 — Proximal Policy Optimization (PPO): The SFT model generates responses, the RM scores them, and PPO updates the policy to maximize reward while constraining divergence from the SFT baseline via a KL penalty term.

A critical design choice in RLHF is the annotation methodology:

• Preference Ranking: Human labelers compare two or more outputs and indicate which is better. This is more reliable than absolute scoring because relative judgments have higher inter-annotator agreement and reduce calibration drift.
• Bradley-Terry Model: The reward model typically uses a Bradley-Terry preference framework, where the probability of preferring response A over B is modeled as a sigmoid of the difference in their scalar rewards.
• Elo-Based Systems: Some implementations, like Anthropic's early work, use Elo ratings derived from head-to-head comparisons to construct a continuous skill ranking across many outputs.

A well-documented failure mode where the policy exploits imperfections in the learned reward model:

• Goodhart's Law: When a proxy measure (the reward model) becomes the optimization target, it ceases to be a good proxy. The policy finds adversarial examples that score highly but are low-quality to humans.
• KL Divergence Constraint: PPO includes a KL penalty term that prevents the policy from drifting too far from the SFT baseline distribution, acting as a regularizer against reward hacking.
• Reward Model Ensembling: Training multiple reward models with different initializations and averaging their scores reduces the variance and exploitability of any single reward signal.

DPO is a recent algorithm that eliminates the need for a separate reward model entirely:

• Implicit Reward: DPO reparameterizes the reward function in terms of the optimal policy, allowing direct optimization on preference data using a binary cross-entropy loss.
• Stability Advantage: By bypassing the reward model training and PPO stages, DPO avoids the instability of adversarial reward hacking and simplifies the pipeline to a single fine-tuning step.
• Mathematical Equivalence: DPO solves the same constrained reward maximization objective as RLHF but does so without explicit reinforcement learning, making it computationally lighter and easier to reproduce.

Scaling alignment beyond human feedback through automated oversight:

• Constitutional AI (CAI): Developed by Anthropic, CAI replaces human preference data with a written constitution of principles. The model critiques and revises its own outputs based on these rules, then is fine-tuned on the self-corrected data.
• RLAIF (RL from AI Feedback): Uses a strong LLM as an automated judge to generate preference labels, reducing the bottleneck and cost of human annotation while maintaining alignment quality.
• Scalable Oversight: Both methods address the fundamental limitation of RLHF — human evaluators cannot reliably judge outputs that exceed their own expertise in complex domains like code generation or advanced mathematics.

Safety alignment can degrade model capability, a phenomenon known as the alignment tax:

• Capability Regression: RLHF-tuned models often perform worse on reasoning benchmarks (e.g., GSM8K, MMLU) compared to their base counterparts because the safety objective competes with the helpfulness objective.
• Over-Refusal: Excessively conservative safety training causes the model to reject benign requests that contain trigger words (e.g., 'How do I kill a process?' in a Linux context).
• Mitigation via Prompt Engineering: System-level instructions can calibrate refusal thresholds, and techniques like context distillation can transfer safety behaviors without the full capability penalty of RLHF.

A trained classifier or regression model that predicts a scalar score representing human preference quality for a given prompt-response pair. In RLHF, the reward model is trained on human comparison data and serves as the proxy reward function that the policy model optimizes against using PPO.

• Architecture: Typically a language model with a value head
• Training data: Pairs of responses ranked by human annotators
• Failure mode: Reward hacking occurs when the policy exploits loopholes in the reward model to achieve high scores without genuine quality improvement

A simpler alternative to PPO for alignment where the model generates multiple candidate responses for each prompt, and the highest-scoring response according to the reward model is selected. This best-of-N approach can be used as a baseline or as the fine-tuning dataset for further supervised training.

• Compute cost: Scales linearly with N (number of samples)
• Quality ceiling: Performance is bounded by the best sample in the generation distribution
• Use case: Often employed in the final refinement stage of alignment pipelines

The curated collection of human judgments that forms the foundation of RLHF. Annotators compare two or more model outputs for the same prompt and indicate which is preferred along dimensions like helpfulness, harmlessness, and honesty. Dataset quality directly determines alignment quality.

• Annotation formats: Binary comparison, Likert scale ranking, or best-worst scaling
• Inter-annotator agreement: Measured via Cohen's kappa to ensure consistency
• Bias risks: Annotator demographics, position bias (preferring the first option), and verbosity bias must be controlled