Reinforcement Learning from Human Feedback (RLHF)

RLHF operates through a sequential pipeline that transforms a raw pre-trained model into a helpful, harmless, and honest assistant.

• Stage 1 - Supervised Fine-Tuning (SFT): The base model is fine-tuned on high-quality demonstration data where human labelers write ideal responses to prompts. This establishes a baseline of desired output formatting and style.
• Stage 2 - Reward Model (RM) Training: Human annotators rank multiple model outputs for the same prompt (e.g., Response A > Response B > Response C). A scalar reward model is trained to predict these preference rankings, effectively encoding human judgment into a differentiable loss function.
• Stage 3 - Proximal Policy Optimization (PPO): The SFT model is further trained using reinforcement learning, where the frozen reward model provides the reward signal. PPO is used to update the policy while constraining divergence from the SFT baseline via a KL penalty term.

The mathematical foundation of the reward model relies on the Bradley-Terry model of paired comparisons. Given a prompt x and two responses y_a and y_b, the probability that a human prefers y_a over y_b is modeled as:

P(y_a > y_b) = exp(r(x, y_a)) / (exp(r(x, y_a)) + exp(r(x, y_b)))

Where r(x, y) is the scalar reward assigned by the model. The reward model is trained via maximum likelihood estimation to minimize the cross-entropy loss between predicted preferences and human labels. This transforms subjective human judgment into a continuous, differentiable reward signal suitable for policy optimization.

A critical failure mode in RLHF is reward hacking, where the policy exploits loopholes in the reward model to maximize reward without actually producing better outputs (e.g., generating nonsensical text that the reward model incorrectly scores highly).

To prevent this, PPO incorporates a Kullback-Leibler (KL) divergence penalty between the current policy and the SFT baseline:

R_total = R_reward - β * KL(π_θ || π_SFT)

This term penalizes the policy for deviating too far from the well-behaved SFT distribution, anchoring the model to human-like language and preventing catastrophic forgetting of basic capabilities.

Direct Preference Optimization is a more recent alternative to the standard RLHF pipeline that eliminates the need for a separate reward model entirely.

• DPO reparameterizes the reward function in terms of the optimal policy, allowing the language model itself to be trained directly on preference data using a binary cross-entropy loss.
• This bypasses the complexity and instability of PPO training, simplifies the pipeline from three stages to two, and avoids the reward hacking problem by directly optimizing for human preferences.
• DPO has been shown to match or exceed PPO-based RLHF performance while being significantly more computationally efficient and easier to implement.

Scaling human feedback is expensive and introduces annotator bias. Reinforcement Learning from AI Feedback (RLAIF) and Constitutional AI address this by using an AI model to generate preference data.

• Constitutional AI: A set of written principles (a constitution) guides the AI in critiquing and revising its own outputs. The model generates self-critiques and refinements based on rules like "choose the response that is least harmful."
• RLAIF: An LLM acts as the judge, comparing responses and generating preference labels, which are then used to train the reward model or directly optimize via DPO.
• These methods dramatically reduce reliance on human annotators while maintaining alignment quality and enabling scalable oversight.

The HHH framework defines the three core axes along which human feedback is collected and models are optimized:

• Helpfulness: The model fulfills the user's stated intent efficiently and accurately. It asks clarifying questions when the prompt is ambiguous and provides actionable, complete information.
• Harmlessness: The model refuses to generate toxic, illegal, unethical, or dangerous content. It recognizes adversarial prompts and avoids reinforcing harmful stereotypes or misinformation.
• Honesty: The model provides calibrated expressions of uncertainty, distinguishes between facts and opinions, and does not fabricate information. It accurately represents the limits of its knowledge.

These dimensions are often in tension (e.g., being overly harmless can reduce helpfulness), requiring careful balancing during training.

The supervised learning process of training a scalar reward model to predict human preferences. Human annotators rank multiple model outputs for the same prompt, and the reward model learns to assign higher scores to preferred completions. This acts as a differentiable proxy for human judgment during the RL phase. Key considerations include inter-annotator disagreement, reward hacking, and the distinction between pointwise (single score) and pairwise (comparative) training objectives.

The standard RL algorithm used in the RLHF pipeline to fine-tune the language model policy. PPO constrains policy updates to a trust region using a clipped surrogate objective, preventing catastrophic forgetting of pre-trained knowledge. The algorithm balances exploration (generating diverse outputs) with exploitation (maximizing the reward signal). A KL divergence penalty is often added to keep the policy close to the supervised fine-tuned reference model.

An alternative to RLHF that eliminates the need for a separate reward model. DPO directly optimizes the language model policy using a binary cross-entropy loss on human preference pairs. This reparameterization maps reward functions to optimal policies, making the training pipeline simpler and more stable. DPO avoids the credit assignment and reward over-optimization challenges inherent in PPO-based RLHF, while achieving comparable alignment performance.

A methodology developed by Anthropic that replaces human feedback with a set of written principles called a constitution. The model generates self-critiques and revisions based on these rules, creating a synthetic preference dataset for RL training. This approach scales alignment without human bottleneck constraints. The process involves two phases: supervised refinement (revising harmful outputs) and RL from AI feedback (training a preference model from constitutional comparisons).

A regularization term added to the RLHF objective that penalizes the policy model for deviating too far from its reference distribution (typically the supervised fine-tuned checkpoint). This prevents mode collapse, language degradation, and preserves the model's general capabilities. The coefficient β controls the trade-off between reward maximization and distributional fidelity. Too little constraint leads to gibberish outputs optimized for the reward model; too much prevents meaningful alignment.

The human annotation pipeline that generates the training signal for reward models. Annotators are presented with a prompt and multiple model responses, then rank them according to criteria like helpfulness, harmlessness, and honesty. Collection protocols must address position bias (ordering effects), length bias (preference for verbose outputs), and demographic representativeness. Platforms like Scale AI and Surge AI provide managed workforces for this task.