Reinforcement Learning from Human Feedback (RLHF)

The initial stage where a pre-trained language model is adapted to a target domain or style using a high-quality dataset of (prompt, response) pairs. This creates a baseline policy model that can generate coherent, on-task outputs before preference alignment begins.

• Purpose: Teaches the model the desired format and task.
• Data: Curated demonstrations from human experts or existing high-quality sources.
• Output: The SFT model serves as the starting point for the RL phase and as the reference for the KL divergence penalty.

A preference learning stage where a separate neural network is trained to predict human preferences. It learns to output a scalar reward score, with higher scores indicating more preferred responses.

• Data: Trained on a preference dataset containing pairs of model outputs where humans have indicated a winner and a loser.
• Objective: Typically uses a loss function based on the Bradley-Terry model, maximizing the likelihood that the preferred response receives a higher score.
• Function: The trained reward model acts as an automated proxy for human judgment during the intensive RL phase.

The core alignment phase where the SFT model (the policy) is optimized using reinforcement learning, with the reward model providing the training signal. Proximal Policy Optimization (PPO) is the standard algorithm.

• Actor-Critic Setup: The policy network (actor) generates text, while a separate value network (critic) estimates the expected future reward to reduce variance.
• Reward Function: Combines the reward model score with a KL divergence penalty to prevent the policy from deviating too far from the safe SFT baseline.
• Challenge: Prone to reward overoptimization (reward hacking) if the reward model has flaws.

The foundational human-annotated data required to train the reward model. It captures subjective human judgments on output quality.

• Format: Typically consists of triples: (prompt, chosen response, rejected response).
• Collection: Human annotators rank or choose between multiple model-generated responses to the same prompt.
• Scale: Quality and breadth are critical; datasets often contain tens of thousands to millions of comparisons (e.g., Anthropic's HH-RLHF).

A crucial regularization term added to the reward function during RL optimization. It penalizes the policy for generating outputs with low probability under the original SFT model.

• Purpose: Prevents mode collapse and catastrophic forgetting of general language capabilities (the alignment tax).
• Mechanism: Measures the difference between the probability distributions of the current policy and the reference SFT policy. A high KL penalty forces the policy to remain close to its initial, coherent state.
• Balance: The strength of the penalty is controlled by a beta (β) hyperparameter, trading off reward maximization against conservatism.

The specific on-policy reinforcement learning algorithm that performs the gradient updates to the language model policy. It is designed for stability with neural network function approximators.

• Key Innovation: Uses a clipped objective function to ensure policy updates are small and stable, enforcing a trust region.
• Components: Involves an actor (policy), a critic (value function), and sometimes a separate language model for forward passes to compute the KL penalty.
• Generalized Advantage Estimation (GAE): Often used with PPO to compute low-variance advantage estimates for better credit assignment.

A core challenge is scalable oversight: designing systems where human supervisors can reliably evaluate outputs from models that may eventually surpass human capabilities. This leads directly to reward overoptimization (or reward hacking), where the policy learns to exploit imperfections in the reward model to achieve a high score while producing outputs misaligned with true human intent (e.g., generating verbose, flattering text that lacks substance). The Kullback-Leibler (KL) divergence penalty is a common, but imperfect, mitigation to constrain policy drift.

The canonical RLHF pipeline is resource-intensive and complex, involving multiple sequential training stages:

• Supervised Fine-Tuning (SFT) to create a baseline policy.
• Training a separate reward model on a high-quality preference dataset.
• Running Reinforcement Learning (typically Proximal Policy Optimization (PPO)) using the reward model. This requires maintaining multiple large models in memory, extensive hyperparameter tuning for stable RL training, and significant compute cycles, making it prohibitive for many organizations.

RLHF's effectiveness is fundamentally bounded by the quality and scale of its human feedback data. Key limitations include:

• Annotation Cost & Consistency: Collecting large-scale, high-quality pairwise preferences from humans is expensive and suffers from inter-annotator disagreement.
• Limited Feedback Granularity: Simple binary preferences provide a weak learning signal compared to rich, detailed critiques.
• Coverage Gaps: The fixed preference dataset may not cover the vast space of possible model inputs and failure modes, leading to poor generalization. Methods like Chain of Hindsight aim to enrich this feedback format.

Alignment tax refers to the observed degradation in a model's general capabilities or performance on broad benchmarks after RLHF fine-tuning. Optimizing heavily for a specific objective like "helpfulness" or "harmlessness" can reduce performance on unrelated tasks like code generation or factual recall. This trade-off forces a difficult balance between alignment and general utility, complicating the deployment of a single, universally capable model.

The reinforcement learning phase of RLHF is notoriously unstable. Challenges include:

• High Variance: Policy gradient methods like PPO can suffer from high variance in updates, leading to erratic training curves.
• Catastrophic Forgetting: The policy may rapidly lose skills learned during pre-training or SFT.
• Hyperparameter Sensitivity: Training stability is highly sensitive to the scale of the KL divergence penalty, learning rates, and batch sizes. Techniques like using a trust region and Generalized Advantage Estimation (GAE) are employed to mitigate these issues but add complexity.

The inefficiency of online RL (PPO) has driven the development of offline RLHF methods like Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), and Kahneman-Tversky Optimization (KTO). These algorithms bypass the need to train a separate reward model and run complex RL, instead deriving a loss function directly from the preference data. Furthermore, Parameter-Efficient Fine-Tuning (PEFT) for RLHF, such as applying Low-Rank Adaptation (LoRA), is critical for reducing the memory footprint and compute cost of alignment, making it more accessible.

A reward model is a neural network trained to predict a scalar reward value for a given language model output. It is the core component that encodes human preferences into a differentiable signal.

• Training Data: Typically learns from a preference dataset of human-ranked output pairs.
• Function: Acts as a proxy for human evaluators during the intensive RL phase, providing fast, automated feedback.
• Critical Challenge: Prone to reward overoptimization, where the policy learns to exploit its flaws, generating high-reward but undesirable outputs.

Direct Preference Optimization (DPO) is an offline alignment algorithm that directly optimizes a language model policy to satisfy human preferences, bypassing the reinforcement learning loop.

• Mechanism: Derives a closed-form loss from the Bradley-Terry model, treating the language model itself as both the policy and the implicit reward function.
• Advantage: Eliminates the need to train and sample from a separate reward model, simplifying the pipeline and reducing computational cost.
• Trade-off: Operates on a fixed dataset, which can limit its ability to explore and improve beyond the provided preferences.

Proximal Policy Optimization (PPO) is an on-policy reinforcement learning algorithm that serves as the standard RL optimizer in the RLHF pipeline.

• Core Innovation: Uses a clipped objective function to constrain policy updates, preventing destructively large changes that collapse performance. This enforces a trust region.
• Architecture: An actor-critic method where the actor (the policy network) generates text and the critic (value network) estimates the expected reward.
• Role in RLHF: The algorithm that performs gradient ascent on the policy, using rewards from the reward model (minus a KL divergence penalty) to steer generation.

The Kullback-Leibler (KL) Divergence Penalty is a critical regularization term added to the reward function in RLHF to constrain the fine-tuned policy.

• Purpose: Prevents the policy from deviating too far from the initial supervised fine-tuned (SFT) model during RL optimization.
• Mechanism: It penalizes the statistical difference between the current policy and the reference SFT model. This is a form of reward shaping.
• Outcome: Mitigates reward overoptimization and mode collapse, ensuring the model retains coherent language capabilities and doesn't 'hack' the reward model with gibberish.

A preference dataset is the foundational collection of human judgments used to train alignment models.

• Standard Format: Consists of triples (prompt, chosen_response, rejected_response), where humans have indicated a preference.
• Usage: Primarily used to train the reward model. Also serves as the direct training data for offline methods like DPO.
• Scalability Challenge: High-quality human annotation is expensive and slow, driving research into Reinforcement Learning from AI Feedback (RLAIF) and Constitutional AI for automated label generation.

PEFT for RLHF involves applying parameter-efficient fine-tuning techniques to the RLHF pipeline, drastically reducing its memory and computational cost.

• Common Technique: Using Low-Rank Adaptation (LoRA) for RLHF to train the actor and critic networks. Instead of updating all 70B+ parameters of a model, LoRA updates only millions of parameters in low-rank matrices.
• Impact: Makes the alignment of massive models feasible for organizations without exascale compute, democratizing access to state-of-the-art aligned models.
• Benefit: Enables efficient multi-task alignment and safer experimentation by isolating and managing the small set of adapted weights.