Reinforcement Learning from Human Feedback (RLHF)

The initial stage where a pre-trained base model is fine-tuned on a high-quality dataset of human-written demonstrations. This dataset typically consists of (instruction, desired response) pairs.

• Purpose: Creates a competent starting policy that understands the target task format and produces coherent, on-topic outputs.
• Mechanism: Standard supervised learning via maximum likelihood estimation (MLE).
• Example: For a helpful assistant model, the dataset would contain thousands of examples where a human has written an ideal response to a user query.

A separate model is trained to predict human preferences, serving as a proxy for a human evaluator. It learns to score responses based on desirability.

• Data Collection: Humans rank multiple model-generated responses to the same prompt, creating pairwise comparison datasets (e.g., Response A is preferred over Response B).
• Architecture: Typically a transformer that takes a (prompt, response) pair and outputs a scalar reward score.
• Loss Function: Uses a comparative loss like the Bradley-Terry model: the model is trained to assign a higher score to the preferred response.

The core alignment stage. The SFT model (the "policy") is optimized using a reinforcement learning algorithm, with the Reward Model providing the reward signal.

• Objective: Maximize the expected reward from the Reward Model while preventing the policy from deviating too far from the SFT baseline (to maintain coherence).
• Algorithm: Proximal Policy Optimization (PPO) is the standard. It performs stable, incremental updates.
• KL Divergence Penalty: A critical component added to the reward function that penalizes the policy for generating text too dissimilar from the SFT model, controlling for reward hacking.

The human-generated data that fuels the Reward Model. Its quality and consistency are paramount to the entire RLHF process.

• Collection Scale: Requires tens of thousands to millions of human judgments.
• Comparison Formats: Can be pairwise (A vs. B), ranked (A > B > C), or absolute scoring.
• Dimensions: Judgments are often made on multiple axes like helpfulness, harmlessness, and truthfulness. Annotator agreement is a key metric.

A more recent and efficient alternative to the traditional RLHF pipeline. DPO directly optimizes the language model policy on preference data without training a separate Reward Model or running complex RL.

• Mechanism: Derives a closed-form solution that re-frames the RL objective as a simple classification loss on the preference data.
• Advantages: Simpler, more stable, and computationally cheaper than PPO-based RLHF.
• Trade-off: While efficient, it may be less flexible for iterative online preference gathering compared to the modular RLHF approach.

A regularization term crucial for preventing reward hacking—where the policy exploits flaws in the Reward Model to maximize score by generating gibberish or repetitive text that the RM incorrectly favors.

• Function: It acts as a "behavioral anchor," keeping the RL-optimized policy's output distribution close to the original SFT model's distribution.
• Mathematical Form: Added to the reward: Total Reward = RM(prompt, response) - β * KL(π_RL || π_SFT) where β is a scaling parameter.
• Outcome: Ensures responses remain natural, diverse, and linguistically coherent while being aligned with preferences.

RLHF is the final, critical step in creating models like ChatGPT and Claude. After initial supervised fine-tuning on instruction-output pairs, RLHF refines the model's behavior based on human preference data. This teaches the model to:

• Generate helpful, harmless, and honest responses.
• Refuse inappropriate requests gracefully.
• Produce outputs that are concise, well-structured, and follow complex instructions. Without RLHF, these models would remain capable but unaligned, often producing verbose, biased, or unsafe outputs.

RLHF is used to align code-specialized LLMs (e.g., GitHub Copilot) by training them on human preferences for correctness, efficiency, and readability. Human labelers compare code snippets, teaching the reward model to prefer:

• Functionally correct code over buggy alternatives.
• Elegant, idiomatic solutions over convoluted ones.
• Well-commented code when appropriate.
• Secure code practices (e.g., avoiding SQL injection patterns). This results in models that generate more reliable, production-ready code and provide higher-quality review comments.

Applying RLHF to content moderation systems allows them to handle nuanced, context-dependent cases that rule-based systems fail on. A reward model is trained on human judgments of toxicity, hate speech, and misinformation, enabling the policy model to:

• Distinguish between educational discussion and promotion of harmful content.
• Understand sarcasm and cultural context.
• Apply proportional moderation actions. This creates more sophisticated, fair, and scalable moderation for social platforms and community forums.

RLHF optimizes summarization models for qualities humans value but are hard to specify with rules. By collecting human comparisons of summaries, the reward model learns to prioritize:

• Faithfulness to the source document (avoiding hallucination).
• Coverage of key points.
• Conciseness and elimination of redundancy.
• Readability for a target audience. This process produces summaries that are not just extractive but truly abstractive and useful, tailored for executive briefings, research paper abstracts, or news digests.

RLHF is a key component in Constitutional AI, a methodology for training harmless AI systems. The process involves:

1. Supervised Learning: The model generates responses to harmful prompts, then critiques and revises them according to a set of principles (a 'constitution').
2. RLHF Phase: A reward model is trained on AI-generated preference data from this supervised stage, creating a 'preference model' for harmlessness.
3. Reinforcement Learning: The policy model is fine-tuned against this preference model. This creates a scalable oversight loop where human input is focused on defining high-level principles, not labeling endless harmful examples.

Direct Preference Optimization (DPO) is a major evolution and simplification of the RLHF pipeline. Instead of training a separate reward model and using complex RL algorithms like PPO, DPO derives a closed-form loss function directly from the human preference data. This allows the policy model to be fine-tuned with standard supervised learning. Key advantages include:

• Training Stability: Eliminates the instability inherent in adversarial RL training.
• Computational Efficiency: Requires less compute and hyperparameter tuning.
• Simplicity: A single training stage replaces the multi-stage RLHF process. DPO has become a popular alternative for achieving high-quality alignment, especially when computational resources or RL expertise are limited.

A reward model is a neural network trained to predict human preferences, serving as a proxy objective in RLHF. It is trained on a dataset of human comparisons between multiple model outputs for the same prompt.

• Training Data: Typically a dataset of triples (prompt, chosen_response, rejected_response) ranked by human labelers.
• Function: During RLHF fine-tuning, this model assigns a scalar reward score to the policy model's generations, guiding the Proximal Policy Optimization (PPO) algorithm.
• Critical Role: The quality and bias of the reward model directly determine the alignment and safety of the final RLHF-tuned model. Reward hacking, where the policy model exploits flaws in the reward model, is a key failure mode.

Proximal Policy Optimization (PPO) is the predominant reinforcement learning algorithm used in the fine-tuning phase of RLHF. It optimizes the language model's policy to maximize the reward signal from the reward model while constraining updates to prevent catastrophic performance collapse.

• Objective: Maximizes expected reward while using a clipped surrogate objective to ensure policy updates are small and stable.
• Challenge in RLHF: Language generation is a high-dimensional, discrete action space, making RL optimization notoriously difficult. PPO's stability is crucial.
• KL Divergence Penalty: A critical component where updates are penalized for straying too far from the original supervised fine-tuned (SFT) model, preserving language fluency and preventing reward hacking.

Instruction tuning is a supervised fine-tuning (SFT) precursor to RLHF, where a base language model is trained on a dataset of (instruction, output) pairs. This teaches the model to follow broad natural language commands, making it more amenable to subsequent alignment via RLHF or DPO.

• Prerequisite for RLHF: RLHF is typically performed on an instruction-tuned model, not the raw pre-trained base. This provides a competent starting policy.
• Dataset Examples: Datasets like ShareGPT, OpenAssistant, and synthetic datasets generated by other models (e.g., Alpaca).
• Distinction from RLHF: Teaches capability (how to follow instructions) rather than preference (which of several capable responses is best). RLHF refines these capabilities based on human taste.

Supervised Fine-Tuning (SFT) is the initial, task-specific training phase that typically precedes RLHF. A pre-trained foundation model is further trained on high-quality, demonstration data relevant to a target domain or style.

• Role in RLHF Pipeline: Creates the initial policy model that RLHF later refines. For chat models, this is often instruction tuning on dialogue examples.
• Data vs. RLHF: Uses demonstration data (ideal responses), whereas RLHF uses preference data (comparisons between responses).
• Limitation: SFT alone cannot optimize for nuanced human preferences where multiple valid responses exist; it simply mimics the provided demonstrations. RLHF addresses this by learning a reward function over the space of possible responses.