Preference Elicitation

The most common data format for modern preference learning. Annotators (human or AI) are presented with two candidate responses to the same prompt and must select their preferred option. This creates a dataset of relative judgments.

• Core Data Structure: Forms the training data for the Bradley-Terry model, the statistical foundation for algorithms like Direct Preference Optimization (DPO).
• Advantages: Forces a clear choice, reducing ambiguity. More reliable than asking for absolute scores.
• Example: Used to create the Anthropic HH-RLHF and OpenAI's WebGPT datasets.

An inference-time and data-collection method where a model generates N candidate responses (e.g., N=4) to a single prompt. A separate reward model or human annotator then ranks all outputs from best to worst.

• Provides Richer Signal: Yields a partial ordering, which is more informative than a single pairwise comparison.
• Efficiency: Allows multiple data points (pairwise preferences) to be inferred from a single ranking task.
• Application: Used in scalable oversight where a supervisor evaluates multiple outputs, and in Rejection Sampling alignment strategies.

A method pioneered by Constitutional AI where an AI model critiques and revises its own responses according to a set of written principles (a 'constitution'). The revisions generate synthetic preferences.

• Process: 1. Generate a response. 2. Use the constitution to prompt a self-critique. 3. Generate a revised response. The (revised, original) pair forms a preference.
• Key Benefit: Dramatically reduces the need for direct human feedback, enabling Reinforcement Learning from AI Feedback (RLAIF).
• Outcome: Produces a dataset where helpful, harmless, and honest responses are preferred.

Elicits preferences by observing optimal behavior, rather than asking for explicit judgments. Inverse Reinforcement Learning (IRL) infers the latent reward function that best explains a set of expert demonstrations.

• Use Case: Ideal for domains where preferences are complex or subconsciously held (e.g., driving, surgical robotics).
• Connection to RLHF: The initial Supervised Fine-Tuning (SFT) stage often uses demonstration data (high-quality responses) to bootstrap a policy before preference-based fine-tuning.
• Challenge: The IRL problem is fundamentally underdetermined; many reward functions can explain the same behavior.

Techniques designed to elicit reliable judgments on tasks too complex for a human to evaluate directly. These methods use AI assistance to amplify human supervisory capacity.

• AI-Assisted Evaluation: A human judges an AI's output with the help of another AI that can explain reasoning or highlight potential flaws.
• Debate: Two AI systems debate the merits of an answer in front of a human judge, who decides the winner. The process reveals information and preferences.
• Goal: Solves the scalable oversight problem, enabling the alignment of superhuman AI systems.

A dynamic approach where preference data is collected in real-time during a model's deployment or training loop, creating a continuous feedback cycle.

• Online Preference Learning: The model's policy is updated based on fresh preferences from its most recent interactions (e.g., user thumbs-up/down in a chat interface).
• Contrasts with Offline: Unlike offline preference learning on a static dataset, this allows the model to adapt to new feedback and correct errors.
• Challenge: Requires robust infrastructure to log interactions, collect labels, and update models safely to avoid catastrophic forgetting or instability.

Preference modeling is the process of training a machine learning model, typically a reward model, to predict human or AI preferences by learning from datasets of ranked or chosen responses. This model acts as a proxy for human judgment.

• Core Function: Maps a given prompt and response pair to a scalar value representing its desirability.
• Training Data: Typically uses pairwise comparison data where one response is chosen over another.
• Application: The trained model's predictions are used as a reward signal for Reinforcement Learning from Human Feedback (RLHF) or to directly rank outputs.

Reward modeling is a specific instantiation of preference modeling focused on creating a function for reinforcement learning. A separate model is trained to predict a scalar reward signal, which is then used to train a policy model via algorithms like Proximal Policy Optimization (PPO).

• Architecture: Often a simple regression head on top of a pre-trained language model.
• Key Challenge: Reward hacking, where the policy model exploits flaws in the reward model.
• Stabilization Techniques: Include reward normalization and using ensemble reward models to improve robustness.

Direct Preference Optimization (DPO) is an algorithm that aligns language models with preferences without training an explicit reward model or running reinforcement learning. It directly optimizes the policy on preference data using a loss derived from the Bradley-Terry model.

• Mechanism: Re-frames the RLHF objective as a supervised loss on preference pairs.
• Advantage: Simpler, more stable, and computationally cheaper than PPO-based RLHF.
• Trade-off: May be less sample-efficient for very large-scale online preference learning.

Pairwise comparisons are the primary data collection method for preference elicitation. Annotators choose between two responses (A and B) to a prompt.

The Bradley-Terry model is the statistical foundation for learning from this data. It assigns a latent 'strength' parameter to each item and models the probability that item A is preferred over item B as a logistic function of the difference in their strengths.

• Data Structure: Forms the core of a preference dataset.
• Mathematical Basis: The DPO loss function is a direct implementation of the Bradley-Terry model's maximum likelihood objective.

Synthetic preferences are AI-generated labels that simulate human judgments, used to create or augment preference datasets. This is crucial for scalable oversight, where human feedback is a bottleneck.

• Generation Methods: Can be created by using a more advanced AI model (a 'critic') to judge a weaker model's outputs, or through Constitutional AI frameworks where a model critiques its own outputs against principles.
• Use Case: Enables Reinforcement Learning from AI Feedback (RLAIF), reducing reliance on expensive human annotation.
• Risk: Can propagate biases or limitations present in the generating model.

These paradigms define how preference data is collected and used during model training.

• Online Preference Learning: The model's policy is updated continuously based on fresh preference data collected from its most recent interactions. This allows adaptation to new feedback but is complex to orchestrate.
• Offline Preference Learning: The model is trained on a static, pre-collected dataset of preferences without further data collection during training. This is simpler and more stable but cannot adapt to new patterns post-deployment.

Hybrid approaches often start with offline learning on a base dataset, followed by limited online updates.