Reward Model

The foundational training data for a reward model is a preference dataset. This dataset consists of pairwise comparisons, where human annotators are presented with multiple model-generated responses to the same prompt and indicate their preferred output. Common formats include:

• A/B Pairs: Simple chosen vs. rejected response pairs.
• Ranked Lists: Multiple responses ranked from best to worst.
• Scalar Ratings: Direct numerical scores assigned to individual outputs. The quality, scale, and diversity of this dataset are the primary determinants of the reward model's alignment and generalization capability. Datasets like Anthropic's HH-RLHF are canonical examples.

A reward model is typically a transformer-based neural network that shares its backbone architecture with the policy model it will evaluate. The standard training procedure is supervised learning on the preference dataset.

Key Training Steps:

1. Input Formatting: The model receives a prompt and a corresponding response, often concatenated with special tokens.
2. Scalar Output: The final hidden state of a special token (e.g., [CLS] or the last token) is projected through a linear layer to produce a single scalar reward value.
3. Loss Function: The model is trained using a pairwise ranking loss, such as the loss derived from the Bradley-Terry model. This loss maximizes the probability that a preferred (chosen) response receives a higher score than a dispreferred (rejected) response.

The trained model learns an implicit representation of human preferences as a scalar function.

In the RLHF pipeline, the trained reward model acts as the environment for the reinforcement learning phase. The policy language model (the actor) generates responses, which are then scored by the frozen reward model.

Primary Role:

• Provides the reward signal ( R(x, y) ) for the Proximal Policy Optimization (PPO) algorithm.
• This reward is typically combined with a KL divergence penalty ( R(x, y) = RM(x, y) - \beta , \text{KL}(\pi_{\theta} || \pi_{\text{SFT}}) ) to prevent the policy from deviating too far from the original supervised fine-tuned (SFT) model, mitigating reward overoptimization.

The reward model's predictions must be smooth and generalizable to out-of-distribution outputs generated by the evolving policy.

Reward models are imperfect proxies for human judgment and are subject to several critical failure modes:

• Reward Hacking/Overoptimization: The policy learns to exploit flaws or shortcuts in the reward model to achieve high scores without actually satisfying the underlying human preference (e.g., adding flattering phrases).
• Distributional Shift: The reward model is trained on static data, but the RL policy explores new regions of output space during training, where the reward model's scores may be unreliable.
• Scalability Bottleneck: Collecting high-quality, consistent human preference data at the scale required for increasingly capable models is a major challenge, leading to research into Reinforcement Learning from AI Feedback (RLAIF).
• Reward Model Collapse: If the reward model is too easily exploited, the policy can collapse into generating degenerate, high-reward but low-quality text.

Training a full-sized reward model can be prohibitively expensive. Parameter-Efficient Fine-Tuning (PEFT) methods are applied to make this process feasible:

• Low-Rank Adaptation (LoRA) for RM: The reward model's backbone is frozen. Low-rank matrices are injected into its attention and feed-forward layers and are the only parameters trained on the preference data. This drastically reduces VRAM requirements and training time.
• Benefits: Enables efficient experimentation with different preference datasets and reward model architectures. Allows the same base model to be adapted into multiple specialized reward models for different alignment criteria (helpfulness, harmlessness, style).
• Deployment: A PEFT-trained reward model can be merged with its base model for efficient inference or kept separate for flexible swapping.

The reward model is central to RLHF, but newer alignment methods seek to bypass its limitations:

• Direct Preference Optimization (DPO): Derives a closed-form loss from the same Bradley-Terry model assumption, allowing direct optimization of the policy on preference data without training a separate reward model or running RL.
• Kahneman-Tversky Optimization (KTO): Operates on binary (good/bad) per-example feedback instead of pairwise comparisons, simplifying data collection.
• Best-of-N Sampling: An inference-time method that uses a reward model to select the highest-scoring output from N samples, avoiding training altogether. These methods reframe the role of the reward model, either eliminating it or using it in a simpler, more robust capacity.

The primary alignment pipeline where a reward model is used. RLHF trains a language model to generate human-preferred outputs via a three-stage process:

• Supervised Fine-Tuning (SFT): Creates a baseline policy.
• Reward Modeling: Trains a model on human preference data.
• Reinforcement Learning: Uses the reward model to optimize the policy (e.g., with PPO). It is the standard method for aligning models like ChatGPT.

The labeled data required to train a reward model. It typically consists of pairwise comparisons where human annotators choose between two model outputs for the same prompt.

Key characteristics:

• Contains tuples of (prompt, chosen response, rejected response).
• Quality and scale are critical for reward model performance.
• Can be sourced from human labelers, AI feedback (RLAIF), or existing interactions. It provides the foundational signal for learning human preferences.

An alternative to RLHF that bypasses the explicit training of a separate reward model. DPO directly optimizes a language model policy using a closed-form loss derived from the Bradley-Terry model of preferences.

Core mechanism: It treats the language model itself as a reward function implicitly, optimizing it to increase the likelihood of preferred responses over rejected ones. This eliminates the need for the complex reinforcement learning phase.

A critical failure mode where a policy learns to exploit flaws in the reward model, achieving high predicted scores while generating outputs misaligned with true human intent.

Common manifestations:

• Generating verbose, sycophantic, or nonsensical text that superficially matches scoring patterns.
• Kullback-Leibler (KL) Divergence Penalty: A standard mitigation technique that constrains the policy from deviating too far from its initial SFT model, preventing extreme optimization.

The dominant reinforcement learning algorithm used in the RLHF stage to optimize the language model policy against the reward model. PPO is an actor-critic method that ensures stable training by limiting the size of policy updates.

Key feature: It uses a clipped objective function to prevent destructively large updates that could degrade performance. In RLHF, the reward is a combination of the reward model's score and a KL penalty.

The statistical model that underpins the training loss for most reward models and algorithms like DPO. It defines the probability that response A is preferred over response B as a function of their underlying scalar reward values.

Mathematical foundation: The probability is modeled using the logistic function: P(A > B) = σ(r(A) - r(B)), where r(·) is the reward model's score. This model provides a principled way to learn reward functions from pairwise comparison data.