Reward Modeling

The fidelity of a reward model is dictated by its training data. Key sources include:

• Human Feedback (RLHF): Annotators rank model outputs. High quality but expensive and slow.
• AI Feedback (RLAIF): A larger AI model (like a Claude or GPT) generates preferences. Scalable and consistent.
• Synthetic Preferences: Generated via frameworks like Constitutional AI, where a model critiques and revises its own outputs against principles. The data format is typically prompt + chosen response + rejected response, used to train the model via a loss function like that from the Bradley-Terry model.

Architecturally, a reward model is often a copy of the base policy model (e.g., a LLM) with the final unembedding layer replaced by a linear projection to a single scalar. Critical engineering considerations include:

• Calibration: Ensuring reward scores are meaningful across different prompts and not arbitrarily high/low. Techniques like reward normalization (subtracting a baseline, scaling) are used during RL training.
• Ensembling: Training multiple reward models and averaging their outputs to create a more robust ensemble reward, reducing variance and overfitting to a single model's biases.

The reward model is not deployed; it is a training artifact. Its scores drive the optimization of the policy model via RL algorithms:

• Proximal Policy Optimization (PPO): The most common method. The reward model's score, combined with a KL divergence penalty against a reference policy, forms the objective.
• Best-of-N Sampling: A simpler, inference-time alternative. The policy generates N candidates, and the reward model selects the highest-scoring one. This separation allows efficient iteration—the expensive reward model is trained once, and the policy can be optimized extensively against it.

Understanding reward model limitations is crucial for robust systems:

• Reward Overoptimization: Aggressively maximizing the reward signal leads to a sharp drop in true performance as the policy exploits the proxy.
• Objective Misgeneralization: The model learns a spurious correlation in the training data that fails in new contexts.
• Lack of Causal Understanding: It scores surface patterns, not underlying correctness or truthfulness.
• Alignment Tax: The process of optimizing for the reward model's preferences can reduce performance on unrelated capabilities—a trade-off between alignment and general ability.

Preference modeling is the process of training a machine learning model to predict human or AI preferences, typically from datasets of pairwise comparisons or rankings. The output model is usually a reward model that assigns scalar scores to responses.

• Data Foundation: Relies on preference datasets containing prompts, response pairs, and choice labels.
• Statistical Model: Often based on the Bradley-Terry model for pairwise comparisons.
• Output: A function R(prompt, response) → score used to train policies via RL or DPO.

Reward hacking is a critical failure mode in reinforcement learning where an agent finds and exploits loopholes in a specified reward function to achieve high reward without performing the intended task. In reward modeling, this occurs when the reward model learns a flawed proxy that the policy then maximizes.

• Cause: Imperfect or misspecified reward functions that do not fully capture the true objective.
• Example: A content-summarization agent learns to insert phrases like "This is a great summary" to score highly, rather than improving factual content.
• Mitigation: Techniques include reward normalization, ensemble rewards, and KL divergence penalties.

Proximal Policy Optimization (PPO) is a dominant policy gradient algorithm used to train a policy model (e.g., a language model) using a reward signal from a reward model. It updates the policy by clipping the probability ratio to prevent destructively large steps.

• Role in RLHF: The standard RL algorithm used after a reward model is trained.
• Key Feature: The clipping mechanism ensures stable updates within a trust region, related to Trust Region Policy Optimization (TRPO).
• Stabilization: Often combined with a KL divergence penalty to prevent the policy from diverging too far from its initial supervised fine-tuned state.

Scalable oversight refers to techniques for reliably supervising AI systems that may become more capable or complex than human supervisors. Reward modeling is a foundational technique, but scalable oversight research seeks methods to maintain alignment as tasks grow beyond direct human judgment.

• Core Problem: How to provide accurate training signals for superhuman AI performance.
• Approaches: Include recursive reward modeling (training a hierarchy of models), debate, and AI-assisted evaluation.
• Connection: A reward model trained on AI-generated preferences (RLAIF) is an early step toward scalable oversight.