Bradley-Terry Model

The model's fundamental operation is the pairwise comparison between two items, labeled i and j. It does not assign absolute scores but defines the probability that item i is preferred over item j. This probability is expressed via the logistic function of the difference between their latent strength parameters.

• Core Equation: P(i > j) = σ(s_i - s_j) = 1 / (1 + exp(-(s_i - s_j)))
• Latent Strengths: Each item has an associated real-valued strength parameter (s_i, s_j). A higher strength indicates a higher probability of being preferred in any matchup.
• Binary Outcome: The model assumes a binary outcome for each comparison: one item wins, the other loses. It is directly applicable to human preference data where annotators choose between two responses (A or B).

The model uses the logistic (sigmoid) function to map the difference in latent strengths to a valid probability between 0 and 1. This choice is not arbitrary; it emerges from the assumption that the observed preference is determined by a latent utility for each item, with the comparison driven by the difference in these utilities plus random noise following a Gumbel distribution.

• Derivation from Random Utility: If the perceived utility of item i is U_i = s_i + ε_i, where ε_i is Gumbel-distributed noise, then the probability P(U_i > U_j) simplifies to the logistic form.
• Interpretability: The strength parameters (s_i) exist on an interval scale. This means their absolute values are not meaningful in isolation, but the differences between them directly dictate win probabilities.
• Identifiability: The model parameters are only identifiable up to an additive constant. Adding the same number to all strength parameters does not change any pairwise probabilities. A constraint (like sum(s)=0 or s_1=0) is typically applied during estimation.

Given a dataset of observed pairwise comparisons, the model's strength parameters are estimated by maximizing the likelihood of the observed data. The likelihood function for a set of independent comparisons is the product of the probabilities for each observed outcome.

• Likelihood Function: L(s) = Π_{(i,j) in wins} P(i > j) * Π_{(i,j) in losses} (1 - P(i > j))
• Log-Likelihood: In practice, the log-likelihood is maximized because it converts the product into a sum, which is numerically more stable: LL(s) = Σ [ y_ij * log(P(i > j)) + (1 - y_ij) * log(1 - P(i > j)) ], where y_ij is 1 if i beat j.
• Optimization: This is a convex optimization problem. Parameters are typically estimated using iterative algorithms like Iterative Scaling or standard gradient-based optimizers (e.g., Adam).

The Bradley-Terry model is not just a standalone statistical tool; it is the theoretical backbone for loss functions in alignment algorithms, most notably Direct Preference Optimization (DPO).

• DPO's Closed-Form Loss: The DPO algorithm derives its loss function by applying the Bradley-Terry model assumption to the outputs of a language model. It posits that the probability a human prefers response y_w over y_l given prompt x is given by: P(y_w > y_l | x) = σ( r(x, y_w) - r(x, y_l) ), where r is an implicit reward function.
• Bypassing Reward Modeling: DPO reparameterizes this reward in terms of the policy (language model) itself and its reference model, allowing direct optimization of the policy without training a separate reward model. The resulting DPO loss is a direct maximum likelihood objective under the Bradley-Terry preference model.
• Foundation for Other Methods: Similar derivations underpin other algorithms like IPO (Identity Preference Optimization) and KTO (Kahneman-Tversky Optimization), which modify or extend the core Bradley-Terry assumption.

The basic Bradley-Terry model has several important generalizations that extend its applicability to more complex preference data structures.

• Plackett-Luce Model: A major generalization for listwise rankings. It defines a probability distribution over full permutations (rankings) of multiple items, not just pairwise comparisons. It is used in alignment scenarios where humans rank several model responses.
• Bradley-Terry with Ties: Extends the model to accommodate draws or 'no preference' outcomes by including a threshold parameter.
• Feature-Dependent Models: Instead of estimating a single strength per item, the strength can be modeled as a linear (or nonlinear) function of item features. This is analogous to moving from a reward model that scores responses to one that scores based on their content attributes.

The model's simplicity relies on specific assumptions that may not always hold in real-world preference data, particularly for language model outputs.

• Independence of Irrelevant Alternatives (IIA): The ratio of probabilities for two items is assumed to be independent of the presence or attributes of any other items. This can be violated if responses are very similar or if a third, clearly terrible option makes two mediocre options seem equally good.
• Transitivity: Implicitly assumed; if A is preferred to B and B to C, then A should be preferred to C. Human preferences can be intransitive.
• Static Strengths: The basic model assumes item strengths are constant, not context-dependent. In LLM alignment, the 'strength' (reward) of a response is highly context-dependent on the prompt (x), which is why the model is applied conditionally (P(y_w > y_l | x)).
• No Modeling of Preference Intensity: The model only captures the binary choice, not how much one item was preferred over the other.

Direct Preference Optimization (DPO) is an offline alignment algorithm that directly optimizes a language model's policy using a loss function derived from the Bradley-Terry model. It bypasses the need to train a separate reward model or perform reinforcement learning.

• Core Mechanism: Uses the Bradley-Terry probability to define a maximum likelihood objective, allowing the language model's implicit reward to be optimized directly.
• Key Advantage: Eliminates the complex and unstable Reinforcement Learning (RL) loop from RLHF, reducing computational cost and training instability.
• Example: Given a preference dataset, DPO adjusts the model's weights so that the likelihood of generating the preferred response increases relative to the dispreferred one, as defined by the Bradley-Terry pairwise comparison.

The Plackett-Luce model is a generalization of the Bradley-Terry model from pairwise comparisons to listwise rankings of multiple items. It defines a probability distribution over all possible permutations of a set of items.

• Mathematical Relationship: The Bradley-Terry model is a special case of the Plackett-Luce model for rankings of just two items.
• Application in AI: Used in alignment methods that leverage ranked lists of multiple model responses, providing a richer training signal than simple pairwise preferences.
• Example: If a human ranks five model outputs from best to worst, the Plackett-Luce model calculates the probability of that specific ranking order based on each output's latent score.

Reinforcement Learning from Human Feedback (RLHF) is a multi-stage alignment pipeline where a reward model, trained on human preference data using the Bradley-Terry model, provides training signals to optimize a language model policy via reinforcement learning.

• Bradley-Terry's Role: The reward model is typically trained using a binary cross-entropy loss based on the Bradley-Terry probability that one response is preferred over another.
• Pipeline Stages: 1) Supervised Fine-Tuning (SFT), 2) Reward Model Training, 3) Policy Optimization (e.g., with PPO).
• Contrast with DPO: RLHF explicitly separates reward modeling and RL, while DPO collapses these steps using the same underlying statistical model.

A reward model is a neural network trained to predict a scalar reward value for a given language model output. It is a critical component in RLHF, where it is trained on datasets of human pairwise comparisons.

• Training Objective: The model is trained to score preferred responses higher than dispreferred ones. The standard loss function is derived from the Bradley-Terry model, maximizing the log-likelihood of the observed preferences.
• Function: Once trained, it acts as a proxy for human judgment, providing reward signals to guide reinforcement learning algorithms like PPO.
• Key Challenge: Reward overoptimization (or "hacking") can occur if the policy learns to exploit imperfections in the reward model, generating high-scoring but undesirable outputs.

A preference dataset is a collection of data used for AI alignment, containing pairs (or rankings) of model outputs where human annotators have indicated a preferred response. It is the foundational training data for the Bradley-Terry model.

• Standard Format: Each data point typically includes a prompt, a chosen (preferred) response, and a rejected (dispreferred) response.
• Usage: This data directly trains reward models in RLHF and is the sole requirement for Direct Preference Optimization (DPO).
• Scale & Quality: The size and consistency of the preference dataset are primary factors in the effectiveness and safety of the resulting aligned model.

The Bradley-Terry model relies on the logistic function (sigmoid) to convert the difference between two latent scores into a probability. The latent score represents the underlying, unobserved quality or utility of an item.

• Core Equation: P(i > j) = σ(s_i - s_j) = 1 / (1 + exp(-(s_i - s_j))), where s_i and s_j are the latent scores for items i and j.
• Interpretation: In alignment, the latent score s is typically the output of a reward model for a given response. The model learns to assign higher latent scores to responses preferred by humans.
• Mathematical Foundation: This formulation transforms a regression problem (predicting a score) into a probabilistic classification problem (predicting a preference), which is more robust for learning from subjective human judgments.