Reinforcement Learning from Human Feedback (RLHF)

RLHF begins by generating a dataset of human-ranked outputs. For a single prompt, the model generates multiple responses (e.g., A and B). Human labelers select the superior response, creating a preference pair. This comparative judgment is psychologically more reliable than asking a human to assign an absolute score. The goal is to capture the subtle, often unspoken, values that define a 'good' response, such as factual accuracy, safety, and contextual relevance.

The collected preference pairs train a separate Reward Model (RM) . This is typically a fine-tuned language model that outputs a scalar score predicting human preference. The RM is trained on a ranking loss: it learns to assign a higher score to the chosen response than the rejected one. This model acts as a proxy for human judgment, providing a fast, differentiable reward signal that replaces the slow, expensive human evaluation loop during the main policy optimization phase.

The core RL algorithm used to fine-tune the language model policy. PPO is favored for its stability. The process involves:

• Policy: The language model generating text.
• Action Space: The vocabulary tokens.
• Reward: The scalar score from the Reward Model. PPO updates the model weights to maximize the reward while using a clipped surrogate objective to prevent the policy from changing too drastically in a single update, avoiding catastrophic forgetting of pre-trained knowledge.

A critical regularization term added to the PPO reward function. The Kullback–Leibler (KL) divergence measures how much the fine-tuned policy has drifted from the original supervised fine-tuned (SFT) model. By penalizing high divergence, RLHF prevents the model from 'hacking' the reward model by generating nonsensical but high-scoring text. This ensures the final model retains coherent language skills and doesn't collapse into a narrow distribution that merely exploits the proxy reward.

An alternative to the standard RLHF pipeline that eliminates the need for a separate Reward Model. DPO reparameterizes the reward function in terms of the optimal policy, allowing the language model to be trained directly on the preference data using a binary cross-entropy loss. This simplifies the pipeline, removes the complexity of PPO tuning, and is more stable because it avoids training a potentially flawed proxy reward model that the policy might later exploit.

A technique that scales oversight by replacing human feedback with a set of written principles (a 'constitution'). The model generates self-critiques and revisions of its own outputs based on these rules. This process generates a synthetic preference dataset for RLHF training. CAI reduces reliance on expensive human labelers and makes the alignment process more transparent and auditable, as the specific rules governing behavior are explicitly defined rather than implicit in a dataset of human clicks.

A distinct model trained to predict human preference scores. It replaces the environment reward signal in classical RL by learning a scalar value from pairwise comparison data.

• Trained on human rankings of model outputs
• Outputs a single score representing 'quality'
• Used as the objective function for policy optimization
• Susceptible to reward hacking if poorly specified

The standard RL algorithm used to update the language model policy in most RLHF implementations. It constrains the policy update size to prevent catastrophic forgetting of pre-trained knowledge.

• Uses a clipped surrogate objective for stability
• Balances exploration with exploitation
• Requires careful KL-divergence penalty tuning
• Often paired with a frozen reference model

An alternative alignment method developed by Anthropic that replaces human feedback with a written constitution of principles. The model generates self-critiques and revisions based on these rules.

• Reduces reliance on human annotator labor
• Uses supervised fine-tuning on self-revised outputs
• Applies RL from AI Feedback (RLAIF) for final refinement
• Principles cover harmlessness, honesty, and helpfulness

A simpler alternative to RLHF that directly optimizes a policy from human preference data without training a separate reward model. It reparameterizes the reward function in terms of the optimal policy.

• Eliminates the need for PPO training loops
• More stable and computationally efficient
• Uses a binary cross-entropy loss on preference pairs
• Mathematically equivalent to RLHF under the Bradley-Terry model

A regularization term added to the RLHF objective that penalizes the policy for deviating too far from the base pre-trained model. This prevents over-optimization of the reward signal.

• Measured between current and reference policy distributions
• Preserves general capabilities during alignment
• Coefficient requires careful hyperparameter tuning
• Too low: reward hacking; too high: no alignment effect

The human annotation process that generates the training signal for RLHF. Annotators compare pairs of model outputs and select the preferred response based on criteria like helpfulness, honesty, and harmlessness.

• Typically uses Likert scales or pairwise ranking
• Requires detailed annotator guidelines
• Inter-annotator agreement metrics measure quality
• Subject to cultural and demographic biases