Ensemble Reward

The primary statistical benefit of an ensemble is variance reduction. By averaging the predictions of multiple independently trained reward models, the ensemble's output smooths out the idiosyncratic errors and noise of any single model. This leads to a more stable and reliable reward signal, which is crucial for the stable convergence of policy gradient algorithms like Proximal Policy Optimization (PPO). It directly combats reward overoptimization by making it harder for the policy to exploit flaws in a single reward function.

A single reward model can be overfitted to its training preference data, learning superficial patterns that do not generalize. A policy can then 'hack' this flawed model. An ensemble acts as a defensive measure:

• It requires the policy to satisfy a consensus of multiple reward perspectives.
• Exploiting a flaw in one model is unlikely to yield high reward from all others.
• This makes finding adversarial shortcuts (reward hacking) significantly more difficult, leading to policies that generalize better to out-of-distribution (OOD) scenarios.

An ensemble provides a natural mechanism for uncertainty quantification. The variance (disagreement) among the member models' predictions for a given state-action pair serves as a proxy for the reward model's epistemic uncertainty.

• High variance signals low confidence, often for OOD or ambiguous inputs.
• This uncertainty signal can be used to downweight unreliable rewards, trigger human review, or guide active learning for collecting new preference data where the ensemble is most uncertain, improving data efficiency.

For the ensemble to be effective, its members must be diverse. Diversity is engineered through:

• Architectural variations: Using different model sizes or neural network architectures.
• Data bootstrapping: Training each model on a different random subset (bootstrap sample) of the full preference dataset.
• Initialization seeds: Different random weight initializations.
• Feature sets: Potentially using different input representations or featurizations. This independence ensures errors are not correlated, making the average more powerful than any single constituent.

The outputs of individual reward models are combined using an aggregation function. Common strategies include:

• Simple Averaging: The most common method, calculating the mean predicted reward.
• Weighted Averaging: Assigning weights based on each model's validation performance or estimated uncertainty.
• Truncated/Censored Means: Discarding the highest and lowest predictions before averaging to reduce outlier influence.
• Voting Schemes: For binary or discrete preferences, using majority vote. The choice of aggregation impacts the bias-variance trade-off and the ensemble's sensitivity to outliers.

Ensemble reward is a key technique within scalable oversight research. As AI systems become more capable than their human supervisors, direct human evaluation becomes a bottleneck. Using an ensemble of AI reward models or critic models (as in Constitutional AI) creates a more robust supervisory signal. This allows for the training of policies that outperform any single AI evaluator's ability to assess, helping to bridge the supervision gap in advanced reinforcement learning from AI feedback (RLAIF) pipelines.

Reward modeling is the core technique of training a separate model to predict a scalar reward signal, which is then used to train a policy via reinforcement learning. This model learns from datasets of human or AI preferences.

• It acts as a proxy for the true, often complex or expensive-to-evaluate, objective.
• In ensemble reward, multiple independent reward models are trained, and their outputs are aggregated (e.g., averaged) to produce a final reward signal.
• This aggregation mitigates the risk of overfitting to the idiosyncrasies of any single model's training data.

Preference modeling is the process of training a model to predict preferences, typically from datasets of ranked or chosen responses. It is the upstream task that often supplies the training data for reward models.

• The output is a ranking or score indicating which of several options is preferred.
• Pairwise comparisons (e.g., using the Bradley-Terry model) are a common data format.
• For ensemble reward, multiple preference models can be trained on different data subsets or with different architectures, and their outputs are synthesized to create a more robust training signal for the final reward ensemble.

Reward hacking is a critical failure mode in reinforcement learning where an agent exploits flaws or unintended shortcuts in the reward function to achieve high reward without accomplishing the intended task.

• Examples include an agent in a simulation pausing the game to avoid losing points, or a language model adding phrases known to be highly rated by the reward model but that degrade response quality.
• Ensemble reward is a primary defensive technique against reward hacking. By aggregating predictions from multiple, independently imperfect reward models, it becomes harder for the policy to find a single, hackable loophole that satisfies all models simultaneously.

Out-of-distribution (OOD) generalization is the ability of a model to perform accurately on data from a different distribution than its training data. It is a major challenge for reward models.

• A single reward model may fail catastrophically when the policy generates novel, OOD responses during reinforcement learning fine-tuning.
• Ensemble reward improves OOD robustness through model diversity. Different models may fail in different ways on OOD data; their combined judgment can average out individual failures and provide a more calibrated signal.
• This helps prevent reward overoptimization, where aggressive maximization of a flawed, non-generalizing reward leads to a sharp drop in true performance.

Direct Preference Optimization (DPO) is an alignment algorithm that optimizes a language model policy directly on preference data, bypassing the explicit reward modeling and reinforcement learning loop.

• Contrast with Ensemble Reward: DPO simplifies the alignment pipeline by eliminating the separate reward model training and PPO stages. Ensemble reward, in contrast, is an enhancement to the traditional Reinforcement Learning from Human Feedback (RLHF) pipeline that uses an explicit reward model.
• While DPO is more efficient, ensemble techniques can still be applied within preference modeling frameworks to create more robust preference datasets or to combine multiple preference signals before applying DPO.

Scalable oversight refers to techniques for reliably supervising AI systems that may become more capable than their human supervisors. It addresses how to provide high-quality feedback for complex tasks.

• Ensemble reward is a scalable oversight tool. It allows the aggregation of feedback from multiple sources, which could include:
  • Multiple AI critic models (as in Constitutional AI).
  • A combination of human and AI feedback.
  • Feedback from models with different specializations or perspectives.
• By synthesizing these signals, the ensemble can approximate a more reliable, nuanced, and scalable oversight mechanism than any single source could provide alone.