Preference Learning

The core mechanism involves training on relative feedback (e.g., "Trajectory A is better than Trajectory B") instead of absolute reward signals. This is often framed as a Bradley-Terry model, where the probability of preferring one outcome over another is modeled via a learned reward function. This approach is more natural for humans, who find it easier to compare outcomes than to assign precise numeric scores.

A primary motivation is to solve the reward specification problem. Manually engineering a reward function for complex tasks (like robotic manipulation) is extremely difficult and often leads to unintended behaviors. By learning the reward directly from human preferences, the system aligns with the user's true, often unquantifiable, objectives, avoiding reward hacking where an agent exploits flaws in a hand-crafted reward.

Efficient preference learning often uses active learning to query human feedback on the most informative comparisons. Instead of passively consuming a fixed dataset, the algorithm selects trajectory pairs where its reward model is most uncertain. This human-in-the-loop design minimizes the amount of costly human feedback required to achieve high performance, making the training process data-efficient.

Preference learning is frequently integrated with Reinforcement Learning (RL) in a paradigm called Preference-based Reinforcement Learning (PbRL). The steps are:

• An RL agent generates trajectories.
• A human provides preference labels on trajectory segments.
• A reward model is trained to predict these preferences.
• The reward model provides synthetic rewards to train the RL agent via standard algorithms like PPO or SAC. This creates a closed-loop system for aligning agent behavior with human intent.

RLHF is the most prominent instantiation of preference learning, famously used to align large language models. The process involves:

1. Supervised Fine-Tuning (SFT) on high-quality demonstrations.
2. Reward Model Training: A separate model learns to score responses based on human preference data.
3. RL Fine-Tuning: The policy is optimized against the reward model using Proximal Policy Optimization (PPO), often with a KL-divergence penalty to prevent excessive deviation from the SFT model. This framework is directly applicable to robotics for learning nuanced tasks.

Preference learning addresses key limitations of standard Imitation Learning (IL):

• Mitigates Compounding Error: Unlike Behavior Cloning, it doesn't just mimic actions but learns an underlying objective, allowing for recovery from unseen states.
• Surpasses the Expert: It can, in principle, learn a reward function that leads to performance superior to the suboptimal demonstrations used to generate comparison queries.
• Scalable Feedback: Collecting preferences is often more scalable than producing perfect expert demonstrations, especially for complex, long-horizon tasks.

Inverse Reinforcement Learning is a technique for inferring a reward function from observed expert behavior, under the assumption that the expert is acting optimally with respect to some unknown reward. While both IRL and Preference Learning aim to recover a human's underlying objective, they differ in input data:

• IRL uses full, optimal demonstrations (state-action trajectories).
• Preference Learning uses comparative feedback (e.g., 'trajectory A is better than B'), which can be easier for humans to provide and is less prone to mimicking suboptimal actions. Preference Learning is often seen as a more scalable and robust alternative to IRL for learning complex reward functions.

Reinforcement Learning from Human Feedback is a prominent application of Preference Learning, most famously used to align large language models. The core process involves:

• Training a reward model on a dataset of human preferences between model outputs.
• Using that reward model to provide training signals for optimizing a policy via reinforcement learning (e.g., Proximal Policy Optimization). In robotics, RLHF adapts this paradigm to physical tasks, where humans provide preferences over robot trajectories or outcomes to shape a reward function for policy optimization.

Direct Preference Optimization is an algorithm that bypasses the explicit reward modeling step of RLHF. Instead of learning a reward model and then using RL, DPO directly optimizes a policy to satisfy human preferences using a closed-form solution derived from the reward modeling objective. Key advantages for robotics include:

• Eliminates the unstable and computationally expensive RL loop.
• Often more stable and data-efficient.
• Enables fine-tuning policies directly on preference datasets, which is valuable when collecting physical demonstrations is costly.

Generative Adversarial Imitation Learning is an adversarial framework where a generator (the policy) learns to produce behavior indistinguishable from expert demonstrations to a discriminator. While GAIL uses full expert trajectories, Preference Learning can be integrated into adversarial frameworks. The key difference is the training signal:

• GAIL's Discriminator: Is trained to classify 'expert vs. generated' state-action pairs.
• Preference-Based Discriminator: Can be trained to predict which of two trajectories a human would prefer, leading to a more nuanced learning signal that focuses on outcome quality rather than exact action mimicry.

A Behavioral Prior is a pre-trained policy or skill library that encodes common, safe, or useful behaviors. Preference Learning is often used to refine or adapt a behavioral prior. The workflow is:

1. A prior policy is learned via offline imitation learning on a broad dataset.
2. The prior is deployed, and a human provides preference feedback on its rollouts.
3. Preference Learning (e.g., via a reward model) is used to fine-tune the prior, specializing it for a specific task or aligning it with nuanced human values without requiring new demonstrations.

Offline Reinforcement Learning involves learning a policy from a fixed dataset of environment interactions without online exploration. Preference Learning is a critical tool for creating usable offline RL datasets from passive human feedback. Typical integration:

• A dataset of suboptimal trajectories is collected (e.g., from scripted policies or human teleoperation).
• Human labelers provide preferences, ranking trajectory segments.
• A reward function is learned from these preferences via Preference Learning.
• An offline RL algorithm (like Conservative Q-Learning or Implicit Q-Learning) is used to train a policy that maximizes the learned reward, effectively distilling human intent from preferences into executable behavior.