Preference Elicitation

A method pioneered by Constitutional AI where an AI model critiques and revises its own responses according to a set of written principles (a 'constitution'). The revisions generate synthetic preferences.

• Process: 1. Generate a response. 2. Use the constitution to prompt a self-critique. 3. Generate a revised response. The (revised, original) pair forms a preference.
• Key Benefit: Dramatically reduces the need for direct human feedback, enabling Reinforcement Learning from AI Feedback (RLAIF).
• Outcome: Produces a dataset where helpful, harmless, and honest responses are preferred.

Elicits preferences by observing optimal behavior, rather than asking for explicit judgments. Inverse Reinforcement Learning (IRL) infers the latent reward function that best explains a set of expert demonstrations.

• Use Case: Ideal for domains where preferences are complex or subconsciously held (e.g., driving, surgical robotics).
• Connection to RLHF: The initial Supervised Fine-Tuning (SFT) stage often uses demonstration data (high-quality responses) to bootstrap a policy before preference-based fine-tuning.
• Challenge: The IRL problem is fundamentally underdetermined; many reward functions can explain the same behavior.

Techniques designed to elicit reliable judgments on tasks too complex for a human to evaluate directly. These methods use AI assistance to amplify human supervisory capacity.

• AI-Assisted Evaluation: A human judges an AI's output with the help of another AI that can explain reasoning or highlight potential flaws.
• Debate: Two AI systems debate the merits of an answer in front of a human judge, who decides the winner. The process reveals information and preferences.
• Goal: Solves the scalable oversight problem, enabling the alignment of superhuman AI systems.

A dynamic approach where preference data is collected in real-time during a model's deployment or training loop, creating a continuous feedback cycle.

• Online Preference Learning: The model's policy is updated based on fresh preferences from its most recent interactions (e.g., user thumbs-up/down in a chat interface).
• Contrasts with Offline: Unlike offline preference learning on a static dataset, this allows the model to adapt to new feedback and correct errors.
• Challenge: Requires robust infrastructure to log interactions, collect labels, and update models safely to avoid catastrophic forgetting or instability.

Direct Preference Optimization (DPO) is an algorithm that aligns language models with preferences without training an explicit reward model or running reinforcement learning. It directly optimizes the policy on preference data using a loss derived from the Bradley-Terry model.

• Mechanism: Re-frames the RLHF objective as a supervised loss on preference pairs.
• Advantage: Simpler, more stable, and computationally cheaper than PPO-based RLHF.
• Trade-off: May be less sample-efficient for very large-scale online preference learning.

Pairwise comparisons are the primary data collection method for preference elicitation. Annotators choose between two responses (A and B) to a prompt.

The Bradley-Terry model is the statistical foundation for learning from this data. It assigns a latent 'strength' parameter to each item and models the probability that item A is preferred over item B as a logistic function of the difference in their strengths.

• Data Structure: Forms the core of a preference dataset.
• Mathematical Basis: The DPO loss function is a direct implementation of the Bradley-Terry model's maximum likelihood objective.

Synthetic preferences are AI-generated labels that simulate human judgments, used to create or augment preference datasets. This is crucial for scalable oversight, where human feedback is a bottleneck.

• Generation Methods: Can be created by using a more advanced AI model (a 'critic') to judge a weaker model's outputs, or through Constitutional AI frameworks where a model critiques its own outputs against principles.
• Use Case: Enables Reinforcement Learning from AI Feedback (RLAIF), reducing reliance on expensive human annotation.
• Risk: Can propagate biases or limitations present in the generating model.

These paradigms define how preference data is collected and used during model training.

• Online Preference Learning: The model's policy is updated continuously based on fresh preference data collected from its most recent interactions. This allows adaptation to new feedback but is complex to orchestrate.
• Offline Preference Learning: The model is trained on a static, pre-collected dataset of preferences without further data collection during training. This is simpler and more stable but cannot adapt to new patterns post-deployment.

Hybrid approaches often start with offline learning on a base dataset, followed by limited online updates.