Preference-Based Reward Learning

The algorithm's primary input is a dataset of human-provided pairwise preferences between short clips of agent behavior, known as trajectory segments. Instead of requiring a human to specify a numeric reward for every state, the system presents queries like "Which of these two outcomes is better?" A reward model (often a neural network) is then trained to predict which segment a human would prefer, learning to assign higher reward to preferred behaviors. This transforms the ill-defined problem of reward specification into a binary classification task.

This method directly addresses the reward specification problem, a major bottleneck in reinforcement learning. Designing a reward function that perfectly captures complex goals (like "cook a meal neatly") is often impossible. Manual rewards can lead to reward hacking, where the agent exploits loopholes. By learning from preferences, the system infers the underlying intent behind the demonstrations, even for tasks where the success criteria are difficult to quantify programmatically. It requires no optimal demonstrations, only relative judgments.

The process is typically iterative and interactive:

• The agent generates new trajectory segments using its current policy.
• A selection strategy (e.g., maximizing information gain or uncertainty) picks the most informative pairs to present to the human.
• The human provides preference labels for these pairs.
• The reward model is updated with the new labels.
• The agent's policy is optimized against the updated reward model (e.g., via RL). This closed-loop allows the system to query about behaviors at the frontier of its current capability, efficiently focusing human attention where it is most needed.

While both IRL and preference-based learning infer a reward function, their data sources differ fundamentally.

• IRL assumes access to complete, optimal (or near-optimal) demonstration trajectories. It solves for a reward function under which the expert's behavior is optimal.
• Preference-Based RL requires only comparative judgments between short clips, which can be suboptimal or exploratory. It does not assume the presented segments are optimal, making data collection simpler and more scalable. IRL aims to explain why an expert chose a specific action, while preference learning asks which of two outcomes is better.

A standard probabilistic model used to translate pairwise comparisons into a learnable reward function. Given two trajectory segments (τ₁, τ₂) and a learned reward function R, the probability that τ₁ is preferred over τ₂ is modeled as: P(τ₁ > τ₂) = exp(Σ R(s,a) in τ₁) / (exp(Σ R(s,a) in τ₁) + exp(Σ R(s,a) in τ₂)) This is equivalent to a logistic regression where the difference in cumulative reward between segments predicts the human's choice. The reward function R is trained to maximize the likelihood of the observed preference data under this model.

This technique is vital for aligning complex AI behaviors with human values in domains where rewards are ambiguous:

• Robotic Manipulation: Teaching a robot to arrange objects "tidyly" or fold clothes without defining every possible crumpled state.
• Autonomous Driving: Learning nuanced driving styles (e.g., assertive vs. cautious) from passenger comfort ratings.
• Content Generation: Fine-tuning large language models or diffusion models to produce outputs preferred by humans, as seen in Reinforcement Learning from Human Feedback (RLHF).
• Game AI: Capturing complex strategic playstyles that are easy to recognize but hard to codify.

Inverse Reinforcement Learning is the foundational problem of inferring an unknown reward function from observations of optimal behavior. The core assumption is that the expert's demonstrated trajectories are optimal with respect to some latent reward function the algorithm aims to recover.

• Mechanism: Given a set of expert trajectories and a model of the environment dynamics, IRL algorithms search the space of possible reward functions to find one that makes the expert's behavior appear optimal.
• Contrast with Preference-Based RL: While IRL typically requires full, optimal demonstrations, preference-based RL uses simpler, comparative human judgments (A vs. B) which are often easier to provide than optimal trajectories.
• Key Challenge: Reward ambiguity—infinitely many reward functions can explain the same expert behavior.

Behavioral Cloning is a supervised learning approach that treats imitation as a regression or classification problem. A policy is trained to directly map observed states to the expert's demonstrated actions.

• Mechanism: The policy network (e.g., a neural network) is trained via supervised learning on a dataset of state-action pairs (s, a) from expert demonstrations.
• Key Limitation: It suffers from covariate shift and compounding errors. Small mistakes cause the agent to visit states not seen in the training data, leading to performance degradation.
• Relation to Preference-Based RL: BC learns a direct policy, while preference-based RL first learns a reward function, which is then used to train a policy via reinforcement learning, often leading to better generalization.

Generative Adversarial Imitation Learning is an adversarial framework for imitation learning. It frames the problem as matching the state-action distribution of the expert without explicitly recovering a reward function.

• Mechanism: A policy (generator) interacts with the environment, and a discriminator is trained to distinguish between state-action pairs from the expert and those from the policy. The policy is trained to 'fool' the discriminator.
• Adversarial Training: The minimax objective directly encourages the policy's occupancy measure to match the expert's.
• Contrast: GAIL performs distribution matching directly on trajectories. Preference-based RL performs distribution matching implicitly via a learned reward function that explains human preferences.

Inverse Optimal Control is the classical control theory counterpart to IRL, focused on deterministic settings. The goal is to infer the cost function that an optimal controller is minimizing.

• Domain: Historically applied to linear-quadratic regulators (LQR), motion planning, and biomechanics.
• Assumption: The demonstrator is assumed to be solving a deterministic optimal control problem.
• Relation: Preference-based reward learning can be viewed as a probabilistic, often model-free extension of IOC, where the 'optimality' criterion is defined by human preferences rather than a mathematically defined cost minimum.

Reinforcement Learning from Human Feedback is a high-level paradigm for aligning AI systems with human intent. It commonly uses preference-based reward learning as its core technical component.

• Pipeline: 1) Collect human preferences on model outputs. 2) Train a reward model on these preferences. 3) Use RL (e.g., PPO) to optimize the policy against the learned reward model.
• Application Scope: While RLHF is famously used to align large language models, the underlying preference learning mechanics are directly applicable to robotics and embodied AI.
• Key Insight: The reward model acts as a differentiable proxy for human judgment, enabling scalable policy optimization.

In preference-based RL, query strategy is critical for sample efficiency. It determines which trajectory segments are presented to the human for comparison.

• Goal: Maximize the information gain about the reward function per human query.
• Common Strategies:
  • Uncertainty Sampling: Query pairs where the reward model's predictions are most uncertain.
  • Information Gain: Choose queries that maximize the expected reduction in reward function entropy.
  • Diversity Sampling: Ensure queried segments cover diverse areas of the state space.
• Impact: A smart query strategy can reduce the number of required human labels by an order of magnitude, making the system practically viable.