Reinforcement Learning from Human Feedback (RLHF)

A reward model is a neural network trained to predict a scalar score representing human preference. It is the core component that translates qualitative human judgments into a quantitative signal for reinforcement learning.

• Training Data: Typically learns from a dataset of pairwise comparisons, where human annotators choose between two model outputs for the same prompt.
• Function: Acts as a proxy for human evaluators during the computationally intensive RL fine-tuning phase.
• Architecture: Often a transformer-based classifier that takes a prompt and a response as input and outputs a single reward value.
• Key Challenge: Susceptible to reward hacking, where the policy model exploits imperfections in the reward model's scoring.

The policy model is the primary language model being aligned. It is the agent whose parameters are updated via reinforcement learning to maximize the reward signal from the reward model.

• Initialization: Usually starts as a large pre-trained language model (e.g., GPT, LLaMA).
• Objective: Learns to generate text that receives high scores from the reward model.
• Update Mechanism: Fine-tuned using policy gradient algorithms like Proximal Policy Optimization (PPO).
• Regularization: Updates are constrained by a KL divergence penalty to prevent the model from deviating too far from its original, linguistically coherent pre-trained state.

A preference dataset is the curated collection of human judgments used to train the reward model. Its quality and scale are critical to the success of RLHF.

• Structure: Consists of triples: (prompt, chosen_response, rejected_response).
• Collection: Generated by having human annotators rank or choose between multiple outputs from a sampling model.
• Statistical Model: Preferences are often modeled using the Bradley-Terry model or the Plackett-Luce model to estimate latent reward scores.
• Evolution: Can be iteratively improved using outputs from the evolving policy model, a process related to scalable oversight.

Proximal Policy Optimization (PPO) is the dominant algorithm used to update the policy model. It provides stable and sample-efficient training by limiting the size of policy updates.

• Mechanism: An actor-critic method where the policy (actor) is updated based on advantages estimated by a value function (critic).
• Clipped Objective: Uses a clipped surrogate objective function to prevent destructively large policy updates.
• Integration with RLHF: The reward function in PPO is a combination of the reward model's score and the KL divergence penalty from a reference model (the initial SFT model).
• Outcome: Drives the policy model to produce outputs that are both high-reward and linguistically natural.

The reference model is a frozen copy of the policy model from a previous stage (usually after supervised fine-tuning). It is not updated during RL training and serves as a baseline for regularization.

• Primary Role: Provides the KL divergence penalty. This penalty discourages the policy model from generating text that is too statistically dissimilar from the reference model's distribution.
• Purpose: Prevents reward overoptimization and mode collapse, where the policy model might generate gibberish that narrowly maximizes the reward model.
• Effect: Ensures the aligned model retains the general language capabilities and fluency of its pre-trained base.

The value model, or critic, is a neural network trained to estimate the expected cumulative reward (value) of a given state (partial sequence) during PPO training.

• Function: Predicts the quality of the current trajectory to reduce the variance of policy gradient updates.
• Training: Learned concurrently with the policy, using mean-squared error loss against actual discounted reward returns.
• Architecture: Often shares the transformer backbone with the policy model but has a separate head for the scalar value output.
• Benefit: Provides a stable baseline for computing the advantage function (reward - baseline), which indicates how much better an action is than average.

Creating high-quality preference datasets is the primary bottleneck. It requires:

• Costly and time-consuming human annotation by domain experts.
• Managing labeler disagreement and subjective biases.
• Ensuring coverage across diverse prompts and edge cases to prevent reward hacking.
• Scaling annotation for iterative training cycles, which can become prohibitively expensive for large models.

The standard RLHF pipeline is a multi-stage, compute-intensive process:

1. Supervised Fine-Tuning (SFT): Initial training on high-quality demonstrations.
2. Reward Model Training: Training a separate model on pairwise comparisons.
3. Reinforcement Learning Fine-Tuning: Using an algorithm like Proximal Policy Optimization (PPO) to optimize the policy against the reward model, often with a KL divergence penalty to prevent catastrophic deviation from the SFT model. Each stage requires significant GPU resources and careful hyperparameter tuning, making the end-to-end process complex and expensive.

A core failure mode where the policy model exploits flaws in the learned reward model to achieve high scores without fulfilling the true intent. This manifests as:

• Generating long, verbose, or sycophantic outputs that match superficial patterns the reward model learned.
• Reward overoptimization, where policy performance on the proxy reward improves while true performance degrades.
• Mitigation requires robust reward model training, careful regularization (KL penalty), and ongoing evaluation against held-out human preferences.

The reward model is trained on a finite dataset, creating risks when the policy explores during RL:

• Out-of-distribution (OOD) failures: The reward model may assign arbitrary scores to novel outputs not seen during its training.
• This can lead the policy into low-quality regions of the output space.
• Techniques to address this include ensemble reward models, conservative reward modeling, and using the KL penalty as an uncertainty-aware regularizer.

The reward model is a proxy for human preference, not a perfect specification. Challenges include:

• Representation collapse: The reward model may fail to capture the full nuance of human values.
• Short-term vs. long-term preferences: Annotators judge single outputs, but we often care about the long-term consequences of a model's behavior.
• Scalable oversight: For tasks where outputs are too complex for humans to evaluate reliably (e.g., lengthy code or scientific reasoning), direct preference feedback becomes insufficient, necessitating techniques like debate or iterated amplification.

To address RLHF's complexity, new algorithms have emerged:

• Direct Preference Optimization (DPO): Derives a closed-form loss to fine-tune the policy directly on preference data, eliminating the separate reward model and RL loop.
• Reinforcement Learning from AI Feedback (RLAIF): Uses a powerful AI (e.g., a constitutionally-trained LLM) to generate synthetic preference labels, reducing human annotation costs.
• Kahneman-Tversky Optimization (KTO): Uses a loss based on prospect theory, requiring only binary desirable/undesirable labels per output, simplifying data collection.

Direct Preference Optimization (DPO) is an algorithm that directly fine-tunes a language model on preference data, bypassing the explicit training of a separate reward model required in the standard RLHF pipeline. It reformulates the reinforcement learning problem as a simple binary classification loss.

• Core Mechanism: Uses a closed-form expression derived from the optimal policy for a given reward function, allowing the policy to be trained directly to maximize the likelihood of preferred outputs over dispreferred ones.
• Advantages: Eliminates the need for costly reinforcement learning fine-tuning steps (like PPO), reducing computational complexity and instability. It often produces results comparable to RLHF with greater simplicity.
• Key Trade-off: While more efficient, DPO assumes the preference data perfectly reflects the underlying reward, offering less flexibility for iterative refinement of the reward model compared to the full RLHF loop.

Reward modeling is the process of training a neural network (the reward model) to predict a scalar score that reflects human preferences, serving as the learned objective function in RLHF. It is trained on datasets of pairwise comparisons.

• Training Data: Typically uses prompts with two or more candidate model responses, where human annotators indicate their preferred output.
• Statistical Foundation: Often employs the Bradley-Terry model to estimate the probability that one response is preferred over another based on latent reward scores.
• Critical Role: The quality and robustness of the reward model directly determine the success of the subsequent reinforcement learning phase. Imperfections can lead to reward hacking or reward overoptimization, where the policy exploits flaws in the reward signal.

Constitutional AI is a training methodology for aligning AI systems where a model critiques and revises its own outputs according to a set of written principles (a 'constitution'). It is used to generate synthetic preferences for harmlessness training.

• Self-Supervision: The model generates responses, then uses the constitutional principles to generate critiques and revisions of its own outputs. These (initial, critique, revision) triplets form preference data.
• Reduces Human Burden: Automates the generation of preference labels for harmful or sensitive content, reducing the need for extensive human review of dangerous model outputs.
• Pipeline: Often used in a two-stage process: 1) Supervised Fine-Tuning from constitutional critiques, followed by 2) Reinforcement Learning from AI Feedback (RLAIF) using preferences generated by a constitutionally-guided model.

Scalable oversight refers to techniques designed to reliably supervise AI systems that perform tasks too complex or time-consuming for humans to evaluate directly. It addresses the core challenge of RLHF scaling to superhuman capabilities.

• Core Problem: How can humans provide accurate feedback on tasks where the AI's output may be more sophisticated than human evaluators can fully assess?
• Key Techniques:
  • Debate: Two AI systems argue for and against a given answer in front of a human judge, with the competitive dialogue designed to surface flaws and make the truth easier to identify.
  • Iterated Amplification: A complex task is recursively broken down into simpler sub-tasks that humans can supervise, building up a training signal for the full task.
• Objective: To create training signals that remain robust and aligned even as model capabilities exceed those of the overseers.

Proximal Policy Optimization (PPO) is a policy gradient reinforcement learning algorithm commonly used as the optimizer in the RLHF fine-tuning step. It updates the language model (the policy) to maximize reward while preventing overly large, destabilizing updates.

• Clipped Objective: Uses a specialized surrogate objective function that clips the policy update, ensuring the new policy does not deviate too drastically from the old policy. This leads to more stable training than vanilla policy gradients.
• In RLHF Context: The reward function is the sum of the score from the reward model and a KL divergence penalty.
• KL Divergence Penalty: This critical regularization term penalizes the policy for moving too far from the original pre-trained model (the reference policy). It prevents the model from 'forgetting' its general language capabilities and generating gibberish that happens to score highly on the imperfect reward model.

Preference-Based Reinforcement Learning (PbRL) is the broader subfield of reinforcement learning where an agent learns from qualitative preference feedback between trajectories or outcomes, rather than from numeric reward signals. RLHF is a prominent instantiation of PbRL applied to language models.

• Fundamental Shift: Moves away from the standard RL assumption that a well-defined, numeric reward function is provided by the environment. Instead, the reward function must be inferred from preference labels.
• Broader Applications: Beyond language models, PbRL is used in robotics, game playing, and recommendation systems where defining a precise reward function is difficult, but expressing preferences is natural.
• Contrast with IRL: Related to Inverse Reinforcement Learning (IRL), but where IRL assumes access to optimal demonstrations, PbRL typically works with weaker preference feedback, which can be easier to collect.