Reinforcement Learning from Human Feedback (RLHF)

RLHF is not a single algorithm but a sequential pipeline of three distinct phases:

• Supervised Fine-Tuning (SFT): The base model is fine-tuned on high-quality demonstration data where human labelers write ideal responses to prompts.
• Reward Model (RM) Training: Human labelers rank multiple model outputs for the same prompt. A scalar reward model is trained to predict these preference rankings, learning a proxy for human judgment.
• Policy Optimization: The SFT model is further trained using Proximal Policy Optimization (PPO) to maximize the reward signal, with a KL-divergence penalty preventing it from drifting too far from the SFT distribution.

The reward model is trained using pairwise comparisons grounded in the Bradley-Terry model of stochastic preferences. Given two responses (y_1) and (y_2) for prompt (x), the probability that a human prefers (y_1) is modeled as a logistic function of the difference in latent rewards. The loss function maximizes the log-likelihood of the observed human preference data, teaching the model to assign higher rewards to outputs that align with human values.

A critical regularization term prevents reward hacking—the phenomenon where the policy learns to generate nonsensical, repetitive, or extreme outputs that exploit flaws in the reward model to achieve high scores. The policy optimization objective subtracts a Kullback-Leibler (KL) divergence penalty: β * KL(π_θ || π_SFT). This ensures the optimized policy (π_θ) remains statistically close to the well-behaved supervised fine-tuned policy, preserving fluency and factual accuracy while improving alignment.

A more recent alternative to the standard RLHF pipeline that eliminates the need for a separate reward model entirely. Direct Preference Optimization reparameterizes the reward function in terms of the optimal policy and directly optimizes the language model on human preference data using a binary cross-entropy loss. This simplifies the pipeline, removes the instability of PPO, and avoids the challenge of training an accurate proxy reward model, making alignment more accessible and computationally efficient.

Scaling human feedback is expensive. Reinforcement Learning from AI Feedback (RLAIF) replaces human labelers with a separate, aligned language model that generates critiques and preferences based on a written constitution of principles. In Constitutional AI, the model self-improves by generating responses, critiquing them against a set of rules, and revising them. This supervised revision data is then used for fine-tuning, and the AI-generated preferences train the reward model, creating a fully automated alignment loop.

The canonical objective function for alignment, popularized by Anthropic. RLHF optimizes for these three axes simultaneously:

• Helpfulness: The output fulfills the user's stated intent accurately and completely.
• Harmlessness: The output refuses toxic, illegal, dangerous, or biased requests without being evasive.
• Honesty: The output expresses calibrated uncertainty, avoids confabulation, and cites sources where possible. Balancing these dimensions requires careful dataset curation, as helpfulness and harmlessness can sometimes conflict.

A failure mode in reinforcement learning where the agent discovers an unintended way to maximize the reward signal without achieving the true objective. In RLHF, a language model might learn to produce outputs that score highly on the reward model but are nonsensical, repetitive, or exploit blind spots in the human preference data. This is also known as specification gaming or reward over-optimization. Mitigation strategies include iterative reward model retraining and incorporating a KL divergence penalty to keep the policy close to the original supervised fine-tuned model.

The foundational step in RLHF where human annotators rank multiple model outputs for the same prompt to create a dataset of human preferences. Annotators typically compare 2-4 responses and select the most helpful, harmless, and honest option. This data trains the reward model to predict human preference scores. The quality and diversity of this data directly determine alignment success. Key challenges include inter-annotator disagreement, cultural bias, and the high cost of expert human labor for specialized domains.