Reward Modeling

A training paradigm where a reward model, trained on AI-generated preference data, provides the reinforcement signal. This scales alignment by reducing reliance on direct human feedback.

• Core Mechanism: An initial LLM generates response pairs; a separate AI (e.g., a more advanced model) ranks them to create a preference dataset for training the reward model.
• Key Advantage: Enables the creation of large-scale, high-quality preference datasets without continuous human-in-the-loop labeling.
• Use Case: Used to fine-tune models for helpfulness and harmlessness when human feedback is scarce or expensive.

The technical challenge of reliably evaluating and guiding AI systems performing tasks too complex for direct human supervision. Reward modeling is a primary tool for this.

• Problem: Humans cannot reliably judge the quality of an AI's work on highly technical or lengthy tasks (e.g., writing complex code, scientific reasoning).
• Solution Approaches: Include Iterated Amplification (breaking complex tasks down into simpler, verifiable subtasks) and Debate (having AI systems argue to surface flaws).
• Role of Reward Models: They act as learned proxies for human judgment, trained on simpler tasks to generalize to harder ones.

A method for aligning AI systems using a set of principles (a "constitution") to guide AI-generated critiques and revisions, which then train a reward model.

• Process: The model generates responses, critiques them according to constitutional principles (e.g., "be helpful, harmless, and honest"), and then revises its own responses.
• Training Data: These AI-generated (critique, revision) pairs form the preference dataset for supervised fine-tuning and subsequent reward model training.
• Outcome: Produces a self-correcting system aligned with high-level principles without needing human feedback on harmful outputs.

The broader machine learning paradigm of training models to predict, optimize for, or generate outputs that align with expressed human preferences. Reward modeling is a specific instantiation.

• Formats: Includes pairwise preferences (choosing A over B), rankings, and numerical scores.
• Key Algorithms: Bradley-Terry models are commonly used to convert pairwise comparisons into a scalar reward function.
• Beyond RLHF: Also used in direct preference optimization methods like DPO, which fine-tune policy models directly on preference data without an explicit reward model.

The problem of inferring the reward function of an agent by observing its optimal behavior. It is the inverse of standard reinforcement learning.

• Contrast with Reward Modeling: IRL typically assumes access to trajectories from an expert policy. Modern reward modeling often uses comparative preferences between outputs, not full trajectories.
• Application: Foundational for apprenticeship learning, where an agent learns to mimic an expert by first recovering the expert's latent reward function.
• Modern Link: Techniques from IRL inform the design of robust preference and reward learning algorithms that can handle ambiguous or partial feedback.

Internal reward signals generated by an agent to encourage exploration and skill acquisition, independent of external task goals. This contrasts with the extrinsic rewards learned by a reward model.

• Examples: Curiosity (reward for predicting error in a learned world model), Surprise, or Empowerment (seeking states with high control over future outcomes).
• Role in RSI: In recursive self-improvement, intrinsic motivation can drive an agent to autonomously seek out novel data or skills that improve its base capabilities.
• Synergy: Can be combined with reward modeling, where an extrinsic reward model aligns the agent with human goals, while intrinsic motivation ensures efficient exploration of the state space.