Reinforcement Learning from Human Feedback (RLHF)

The initial phase where a pre-trained foundation model is adapted to a specific domain or style using a high-quality dataset of human-written demonstrations. This creates a policy model that serves as the starting point for alignment.

• Purpose: Teaches the model the desired format and basic task competency.
• Process: Standard supervised learning on (prompt, ideal response) pairs.
• Outcome: A model capable of generating coherent, on-task outputs, but not yet optimized for human preference.

A preference model is trained to predict which of two model outputs a human would prefer. This model learns a scalar reward function that encodes human values.

• Data Collection: Humans rank or choose between multiple outputs for the same prompt, creating a dataset of pairwise comparisons.
• Architecture: Typically a transformer that takes a (prompt, response) pair and outputs a scalar score.
• Loss Function: Uses a Bradley-Terry model or similar to learn from preference rankings. The trained reward model acts as a proxy for human judgment during the next phase.

The policy model (from SFT) is optimized using a Reinforcement Learning algorithm, with the reward model providing feedback. The goal is to maximize reward while staying close to the original policy to prevent degradation.

• Algorithm: Proximal Policy Optimization (PPO) is commonly used for its stability.
• Objective: Maximize expected reward while constraining the policy change via a KL divergence penalty.
• Challenge: Avoiding reward hacking, where the policy exploits flaws in the reward model to generate high-scoring but nonsensical outputs.

A critical regularization term added to the RL objective during the PPO phase. It prevents the policy model from deviating too far from its initial SFT model distribution.

• Purpose: Maintains generation diversity, prevents mode collapse, and avoids catastrophic forgetting of language capabilities.
• Mechanism: Adds a penalty proportional to the Kullback–Leibler divergence between the current policy and the reference SFT policy.
• Effect: Balances maximizing reward with preserving the natural, coherent language learned during pre-training and SFT.

An alternative algorithm to the traditional RLHF pipeline. DPO directly optimizes a language model policy on preference data using a closed-form loss derived from the same Bradley-Terry model, eliminating the need to train a separate reward model or run PPO.

• Advantage: Simpler, more stable, and often more computationally efficient than the RLHF loop.
• Mechanism: Treats the language model itself as an implicit reward function, optimizing it directly to increase the likelihood of preferred responses over dispreferred ones.
• Relation to RLHF: Provides the same theoretical optimum as RLHF under the preference model, but via a different, more direct optimization path.

Production RLHF systems often operate as an iterative loop, not a one-off process. New model generations are evaluated to create fresh preference data, continuously improving both the reward and policy models.

• Process: Deploy model → collect new human comparisons on its outputs → retrain reward model → fine-tune policy.
• Challenge: Managing distributional shift as the policy model generates outputs different from those in the original training data.
• Outcome: Creates a data flywheel where model improvement drives better data collection, which in turn drives further improvement.

RLHF's performance is fundamentally limited by the quality, scale, and consistency of its human preference data. Key challenges include:

• Scalability Bottleneck: Manually labeling thousands to millions of comparison pairs is slow and expensive.
• Labeler Disagreement: Different annotators may have conflicting preferences, introducing noise into the reward model's training signal.
• Coverage Gaps: It's impossible to label all possible model outputs, leaving the reward model to generalize, sometimes poorly, to unseen scenarios.
• Solution Trends: Many teams now use synthetic data (generated by a teacher model) or AI-assisted labeling to scale data creation, but this can introduce bias.

The policy model can learn to exploit flaws in the reward model's scoring function, a phenomenon known as reward hacking or Goodhart's law. This leads to behaviors that maximize the reward score but degrade actual output quality.

• Examples: Generating long, verbose text to trigger positive sentiment keywords, or inserting phrases known to be highly rated by the reward model, regardless of relevance.
• Mitigation: Requires robust reward model regularization (e.g., weight clipping, dropout), KL divergence penalties to prevent the policy from straying too far from its SFT baseline, and continuous monitoring of reward score drift versus human evaluation.

RLHF is a multi-stage pipeline, each with its own infrastructure demands:

1. Supervised Fine-Tuning (SFT): Requires a high-quality demonstration dataset.
2. Reward Model Training: Involves training a separate model (often a modified version of the SFT model) on comparison data.
3. RL Fine-Tuning: Running Proximal Policy Optimization (PPO) or similar algorithms is computationally intensive and unstable, requiring careful hyperparameter tuning.

• Memory Overhead: The pipeline often requires hosting four models simultaneously: the policy, the reward model, a reference model (for KL penalty), and sometimes a critic model.
• Tooling: Requires mature MLOps for experiment tracking, model versioning, and pipeline orchestration.

During RL fine-tuning, the policy model's output distribution shifts away from the natural language distribution it learned during pre-training and SFT. This can cause:

• Mode Collapse: The model loses linguistic diversity, producing repetitive or generic responses.
• Degradation of General Capabilities: Over-optimization for the reward signal can degrade performance on unrelated but valuable skills (e.g., code generation, creative writing).
• The KL Divergence Penalty is the primary guardrail against this, but tuning its strength is a critical and delicate balance.

Measuring the success of RLHF is non-trivial, as the goal—alignment with nuanced human preferences—is inherently subjective.

• Automated Metrics (e.g., BLEU, ROUGE) are poorly correlated with human judgment of helpfulness and harmlessness.
• Human Evaluation remains the gold standard but is expensive and slow, hindering rapid iteration.
• Emergent Benchmarks: The field relies on proxy benchmarks like MT-Bench (for multi-turn dialogue) or HellaSwag (for commonsense reasoning), but these may not capture the full spectrum of desired behaviors.
• Trade-off Tension: Often there is a measurable trade-off between helpfulness (optimized by RLHF) and truthfulness/hallucination reduction, which must be explicitly managed.

Due to RLHF's complexity, several alternative alignment methods have gained prominence, particularly for smaller teams or models:

• Direct Preference Optimization (DPO): A stable, single-stage algorithm that directly optimizes a policy on preference data without training a separate reward model or using RL. It's simpler and less computationally demanding.
• Reinforcement Learning from AI Feedback (RLAIF): Uses a powerful LLM (like GPT-4) to generate the preference labels, bypassing human labelers. This scales more easily but transfers the bias of the labeling LLM.
• Constitutional AI: Aims to train models to critique and revise their own outputs according to a set of principles (a constitution), reducing reliance on extensive human feedback. These methods address specific RLHF challenges but come with their own trade-offs.

Direct Preference Optimization is an alignment algorithm that fine-tunes a language model policy directly on a dataset of human preferences, eliminating the need for a separate reward model and the complex reinforcement learning loop used in RLHF. It reformulates the RLHF objective as a maximum likelihood problem using a closed-form expression derived from the Bradley-Terry model.

• Core Mechanism: Directly optimizes the policy using a loss function that maximizes the likelihood of preferred completions over dispreferred ones.
• Key Advantage: Simpler, more stable training than RLHF's Proximal Policy Optimization (PPO) stage, often requiring less compute and hyperparameter tuning.
• Trade-off: While efficient, DPO assumes the preference data perfectly reflects the optimization target, whereas RLHF's reward model can generalize to new, unseen prompts.

A Reward Model is a neural network trained to predict a scalar reward, representing human preference, given a prompt and a model-generated response. It is the critical component in RLHF that quantifies alignment.

• Training Data: Trained on datasets of human comparisons, where annotators choose between two model outputs for the same prompt.
• Function: Serves as a proxy for human judgment during the reinforcement learning phase, providing the reward signal used to fine-tune the policy model (e.g., via PPO).
• Architecture: Typically a transformer initialized from the SFT model, with a linear projection head that outputs a single scalar value.

Proximal Policy Optimization is the reinforcement learning algorithm most commonly used in RLHF to fine-tune the policy model against the learned reward model. It optimizes the policy to maximize expected reward while preventing updates that are too large and destabilizing.

• Core Objective: Maximizes a clipped surrogate objective function, ensuring policy updates are 'proximal' (small).
• In RLHF Context: The policy model generates responses, the reward model scores them, and PPO uses these scores to update the policy parameters.
• Challenges: Requires careful tuning and is computationally intensive. Often paired with a KL divergence penalty to prevent the policy from deviating too far from the original SFT model, preserving language quality.

Supervised Fine-Tuning is the initial, mandatory stage of RLHF where a pre-trained base language model is further trained on a high-quality dataset of (prompt, desired response) pairs. This creates a skilled initial policy.

• Purpose: Teaches the model to generate coherent, helpful, and harmless responses in a dialogue or instruction-following format.
• Role in RLHF: The SFT model serves as the initial policy for reinforcement learning and is often used to initialize the reward model. It provides a crucial performance and behavioral baseline.
• Contrast with RLHF: SFT optimizes for mimicking a static dataset, while subsequent RLHF stages optimize for a learned preference function, which can yield more nuanced and human-preferred behaviors.

Instruction Tuning is a form of supervised fine-tuning where a model is trained on a diverse collection of tasks formatted as natural language instructions and their corresponding outputs. It is a common method to create the SFT model used at the start of the RLHF pipeline.

• Goal: Improves the model's zero-shot and few-shot generalization to unseen tasks by teaching it to follow task descriptions.
• Dataset Example: (Instruction: 'Summarize this article:', Input: <article text>, Output: <summary>).
• Relation to RLHF: A high-quality instruction-tuned model is the typical starting point for RLHF. The alignment process (RLHF) then refines the model's behavior within its instruction-following capability based on human preferences.

Constitutional AI is an alternative alignment methodology developed by Anthropic. It uses AI-generated feedback based on a set of governing principles (a 'constitution') to train models to be harmless and helpful, reducing reliance on direct human labeling.

• Process: Involves a supervised stage where the model critiques and revises its own responses based on constitutional principles, and a reinforcement learning stage where it learns from AI feedback.
• Key Difference from RLHF: Replaces human feedback on output preferences with AI-generated feedback based on principled critiques. This aims to scale alignment and make the model's values more explicit and auditable.
• Outcome: Models like Claude are trained using this method, which is seen as a complementary approach to RLHF for achieving robust alignment.