Reinforcement Learning from Human Feedback (RLHF)

RLHF is not a single algorithm but a structured pipeline. It typically involves three sequential phases:

• Supervised Fine-Tuning (SFT): A base pre-trained model is first fine-tuned on high-quality instruction-response pairs to establish competent instruction-following.
• Reward Model Training: A separate model is trained to predict a scalar reward that reflects human preferences, using datasets of pairwise comparisons where humans rank multiple model outputs.
• Reinforcement Learning Fine-Tuning: The SFT model is further optimized using a policy gradient algorithm like Proximal Policy Optimization (PPO), with the reward model providing the training signal.

The core innovation of RLHF is replacing a handcrafted reward function with one learned from human judgments. A reward model is trained as a classifier on datasets where humans indicate which of two (or more) model completions they prefer for a given prompt. The model learns to assign higher scores to outputs that align with nuanced human values like helpfulness, honesty, and harmlessness. This reward model then acts as a proxy for human evaluators during the intensive RL training loop.

The final RL stage treats the language model as a policy that generates sequences of tokens. The Proximal Policy Optimization (PPO) algorithm is used to update the model's parameters to maximize the expected cumulative reward from the reward model, while using constraints to prevent the policy from deviating too far from its original SFT state. This prevents catastrophic forgetting of language capabilities. A KL divergence penalty is a critical component, ensuring generated text remains coherent and on-distribution.

The primary objective of RLHF is value alignment—steering a model's behavior to be more useful and safer according to broad human preferences. Unlike supervised fine-tuning which teaches what to say, RLHF teaches how to say it in a preferred manner. It addresses the alignment problem where a model trained purely to predict the next token may generate plausible but harmful, evasive, or unhelpful content. RLHF directly optimizes for these qualitative, hard-to-specify attributes.

Implementing RLHF at scale introduces significant engineering and research hurdles:

• Reward Hacking: The policy may exploit flaws in the reward model, generating outputs that score highly but are nonsensical or contain hidden negative content.
• Distributional Shift: The data distribution of outputs during RL training drifts from the SFT baseline, requiring careful regularization.
• High Computational Cost: The pipeline requires training multiple large models (SFT, Reward, PPO) and running complex, unstable RL loops, demanding immense GPU resources.
• Human Data Bottleneck: Collecting high-quality, consistent human preference data is expensive and slow, creating a scalability limit.

RLHF exists within a broader ecosystem of alignment techniques:

• Direct Preference Optimization (DPO): A more stable alternative that derives a closed-form loss from the same preference data, bypassing the need for a separate reward model and the complex PPO loop.
• Constitutional AI: An Anthropic methodology where a model critiques and revises its own outputs against a set of principles, reducing direct human feedback.
• Rejection Sampling & Best-of-N: Simpler baseline where the SFT model generates many outputs, and the one with the highest reward model score is selected, though it doesn't update the underlying model.

The reward model (RM) is a critical component trained to predict a scalar score representing human preference. It is typically a smaller transformer model (e.g., 6B parameters for a 70B policy model) trained on a dataset of human preference comparisons. The training uses a Bradley-Terry model to learn from pairs of model outputs where humans have indicated which is better. This RM replaces human labelers in the RL loop, providing fast, scalable feedback signals to guide the policy model's optimization.

The policy model is the base language model being aligned (e.g., an instruction-tuned model). It is optimized using the Proximal Policy Optimization (PPO) algorithm, a stable RL method. PPO updates the policy to maximize the reward signal from the RM while minimizing deviation from the original model (via a KL divergence penalty) to prevent catastrophic performance degradation. The optimization objective is: Maximize E[R(x, y) - β * KL(π_θ || π_ref)], where π_θ is the current policy and π_ref is the reference (SFT) model.

The reference model is a frozen copy of the Supervised Fine-Tuned (SFT) model that serves as a behavioral anchor during RL training. Its primary role is to compute the KL divergence penalty against the current policy model's outputs. This penalty acts as a regularizer, preventing the policy from drifting too far into regions of output space that might exploit the reward model with gibberish or unnatural text that scores highly but is undesirable. It enforces conservatism and maintains language quality.

Direct Preference Optimization (DPO) is an alignment algorithm that directly fine-tunes a language model using a dataset of human preferences, bypassing the need to train a separate reward model. It derives a loss function from the Bradley-Terry model of pairwise comparisons, optimizing the policy to increase the probability of preferred outputs over dispreferred ones.

• Mechanism: DPO uses a closed-form expression for the optimal policy under the reward model, allowing the reward function to be implicitly defined by the policy itself.
• Advantages: Eliminates the unstable and complex reinforcement learning phase of RLHF, reducing computational cost and implementation complexity.
• Use Case: A popular alternative to RLHF for preference alignment, especially when computational resources for reward modeling and PPO are limited.

Reward modeling is the process of training a separate neural network to predict a scalar reward that reflects human preferences. It is the critical second stage in the RLHF pipeline.

• Training Data: Typically trained on datasets of pairwise comparisons, where humans have labeled which of two model responses is better for a given prompt.
• Function: The trained reward model assigns a score to any model-generated response, providing the training signal for the subsequent reinforcement learning phase.
• Key Challenge: Reward hacking, where the policy model learns to exploit flaws in the reward model to achieve high scores without actually improving output quality.

Supervised Fine-Tuning (SFT) is the initial stage in a standard RLHF pipeline. It adapts a base pre-trained language model to a specific domain or instruction-following format using a cross-entropy loss.

• Purpose: Creates a competent initial policy model that can generate coherent, on-task responses before preference alignment begins.
• Training Data: Uses high-quality datasets of instruction-response pairs.
• Relation to RLHF: Provides the starting point for the Proximal Policy Optimization (PPO) phase. The SFT model is often used to generate the responses that humans compare for reward model training.

Proximal Policy Optimization (PPO) is the reinforcement learning algorithm most commonly used to fine-tune the language model in the final stage of RLHF. It optimizes the model's policy to maximize the expected reward from the reward model.

• Objective: Updates the model's parameters to increase the probability of generating high-reward responses while preventing updates that are too large and destabilizing.
• KL Divergence Penalty: A crucial component that penalizes the policy for straying too far from the original SFT model, preventing degradation of language quality and catastrophic forgetting.
• Complexity: This phase is computationally intensive and requires careful hyperparameter tuning to maintain stability.

Constitutional AI is an alignment methodology developed by Anthropic that uses AI feedback based on a set of principles (a "constitution") to train models, reducing direct reliance on human feedback.

• Process: Involves a supervised learning stage where a model critiques and revises its own harmful responses according to constitutional principles, and a reinforcement learning stage where the model is trained to prefer constitutional responses.
• Contrast with RLHF: Aims to create more scalable and principled alignment by encoding values explicitly into rules, rather than learning them implicitly from human preference datasets.
• Outcome: Designed to produce models that are helpful, honest, and harmless (HHH) through self-critique and iterative refinement.

Instruction tuning is a broad supervised fine-tuning process where a model is trained on datasets of instruction-response pairs to improve its ability to understand and follow natural language commands.

• Foundation for RLHF: The model resulting from instruction tuning (often equivalent to the SFT model) is the typical starting point for the RLHF process.
• Goal: Teaches the model the format and scope of responding to instructions, while RLHF and related methods teach the model the quality and preference alignment of those responses.
• Datasets: Examples include Alpaca, ShareGPT, and Dolly, which contain diverse user queries and high-quality assistant replies.