Preference Elicitation

The most foundational query format, where a human is presented with two options (A/B) and selects their preferred one. This method is cognitively simple and directly provides the data structure (winner/loser pairs) used to train Bradley-Terry or Plackett-Luce models.

• Key Use: Training reward models for RLHF and DPO.
• Challenge: Requires many comparisons for high confidence; can be inefficient for ranking many items.

A generalization of pairwise comparison where a human ranks multiple (N) options from most to least preferred. This provides more information per query but increases cognitive load.

• Model Used: Typically analyzed with the Plackett-Luce model.
• Efficiency: More data-efficient than sequential pairwise queries for establishing a full ranking.
• Application: Used in advanced preference dataset creation where annotators evaluate several model generations for a single prompt.

The user provides an absolute rating (e.g., 1-5 stars) or binary label (good/bad) for a single output. This is simpler to collect but introduces bias as ratings are not calibrated across different queries.

• Algorithm Link: Forms the basis for Kahneman-Tversky Optimization (KTO), which only requires binary desirable/undesirable labels.
• Advantage: Lower cost and faster than comparative judgments.
• Disadvantage: Less informative for learning a consistent utility function across different contexts.

An algorithmic strategy where the system intelligently chooses which comparison or question to ask next, aiming to maximize information gain about the user's preferences.

• Goal: Reduce the number of queries needed for an accurate model.
• Methods: Includes uncertainty sampling (query where the reward model is most uncertain) and expected utility of information.
• Relation: A core component of active learning for streams in continuous learning systems.

Used specifically in Preference-Based Reinforcement Learning (PbRL), where a human compares two full trajectories (sequences of states and actions) instead of single outputs. This elicits preferences over behaviors or strategies.

• Utility: Essential for tasks where defining a step-by-step reward function is impossible, but overall outcomes can be judged.
• Challenge: Comparing long trajectories is cognitively demanding for humans, leading to scalable oversight challenges.

Advanced strategies for complex tasks where direct preference is hard to judge. In Debate, two AI systems argue over an answer, and the human judges the debate to reveal the better underlying argument. Iterated Amplification breaks a complex question into simpler sub-questions for human evaluation.

• Purpose: Address scalable oversight for superhuman AI systems.
• Outcome: Elicits preferences over reasoning processes, not just final answers, aligning with process supervision methodologies.

RLHF is the foundational technique for aligning language models using human preferences. It involves a three-stage pipeline:

• Train a reward model on a dataset of human pairwise comparisons.
• Use the reward model to provide scalar feedback to a policy model (e.g., via Proximal Policy Optimization).
• Apply a KL divergence penalty to prevent the policy from deviating too far from its original, helpful behavior. This process directly translates elicited preferences into optimized model behavior.

DPO is an algorithm that simplifies the RLHF pipeline. Instead of training a separate reward model and running reinforcement learning, DPO treats the preference learning problem as a direct classification loss on the policy network itself. It derives a closed-form solution linking the optimal policy to the reward function implied by the Bradley-Terry model, enabling stable and efficient fine-tuning directly on preference data.

Reward modeling is the process of training a neural network to predict a scalar value that reflects human preferences. It is the critical intermediary step in RLHF.

• Training Data: Typically uses pairwise comparisons (e.g., Response A vs. Response B).
• Statistical Model: Often uses the Bradley-Terry model to estimate the probability one response is preferred.
• Output: A reward model that can score any model output, providing the training signal for the subsequent reinforcement learning phase.

These are critical failure modes in preference-based systems. Reward hacking occurs when a model exploits loopholes in the learned reward function (e.g., generating verbose, flattering text to maximize a reward model trained on 'helpfulness'). Reward overoptimization is the phenomenon where further optimization of a proxy reward (the reward model's score) leads to a decrease in performance according to the true objective (human judgment), due to imperfections and biases in the reward model.

These are methods for generating preference data without direct human labeling for every comparison. Constitutional AI involves a model critiquing and revising its own outputs according to a set of written principles. Reinforcement Learning from AI Feedback (RLAIF) uses a separate AI (like a large language model) to generate the preference labels for training the reward model. Both aim to create synthetic preferences to scale oversight and reduce reliance on human annotators.

A family of methods to supervise AI systems performing tasks too complex for direct human evaluation, closely related to advanced preference elicitation.

• Debate: Two AI systems argue for and against an answer to help a human judge discern the truth.
• Iterated Amplification: A complex task is recursively broken into simpler sub-tasks that humans can supervise.
• Process Supervision: Providing feedback on the intermediate reasoning steps, not just the final output. These techniques aim to elicit reliable preferences and judgments on highly complex outputs.