Reinforcement Learning from Human Feedback (RLHF)

The Supervised Fine-Tuning (SFT) Model is the initial, instruction-tuned version of the base pre-trained model (e.g., GPT, LLaMA). It is trained on a high-quality dataset of human-written demonstrations for the target task, which provides a strong behavioral prior.

• Purpose: Establishes competent, coherent behavior before preference learning begins.
• Process: Uses standard maximum likelihood estimation (MLE) on prompt-response pairs.
• Output: A model capable of generating relevant, high-quality outputs that serve as the starting policy for the subsequent reinforcement learning phase.

The Reward Model (RM) is a crucial proxy trained to predict human preferences. It learns to score model outputs, transforming qualitative human judgments into a quantitative reward signal.

• Training Data: Created from pairwise comparison data, where human labelers choose which of two model responses is better for a given prompt.
• Architecture: Typically a copy of the SFT model with its final layer replaced by a regression head that outputs a scalar reward score.
• Loss Function: Trained using a Bradley-Terry model or similar, which models the probability that one response is preferred over another.
• Function: During RL fine-tuning, it provides the reward r(x, y) for a prompt x and generated response y, guiding the policy model.

The Policy Model is the final, aligned model produced by applying reinforcement learning to the SFT model, using the Reward Model to guide updates.

• Algorithm: Typically uses Proximal Policy Optimization (PPO), a policy gradient method stable for fine-tuning large language models.
• Objective: Maximizes the expected reward from the RM while minimizing deviation from the SFT model via a KL-divergence penalty. This prevents the model from exploiting the RM's weaknesses and generating gibberish that scores highly.
• Process: The model (the actor) generates responses, which are scored by the RM (the critic). PPO uses these scores to compute policy gradients and update the model's parameters.

The Human Preference Dataset is the foundational labeled data required to train the Reward Model. Its quality and scale directly determine the alignment performance of the final system.

• Collection: Labelers are presented with a prompt and 2-4 responses generated by the SFT model. They rank the responses based on criteria like helpfulness, harmlessness, and factual accuracy.
• Format: Stored as tuples (prompt, chosen_response, rejected_response).
• Challenges: Requires careful labeler training and quality control to ensure consistent, high-quality judgments. Scaling this data collection is often the primary bottleneck in RLHF.

The KL-Divergence Penalty is a regularization term added to the RLHF reinforcement learning objective. It is critical for maintaining model stability and preventing reward hacking.

• Problem: Without constraint, the policy model can over-optimize the Reward Model, generating outputs that get a high reward score but are nonsensical or degenerate (e.g., repetitive phrases).
• Solution: The penalty discourages the policy from moving too far from the probability distribution of the initial SFT model. The full RL objective becomes: Objective = E[r(x, y)] - β * KL(π_RL(y|x) || π_SFT(y|x)), where β is a hyperparameter controlling the strength of the penalty.
• Effect: Ensures the model remains coherent and linguistically natural while adapting to human preferences.

Proximal Policy Optimization (PPO) is the default reinforcement learning algorithm used to fine-tune the policy model against the Reward Model in RLHF. It is chosen for its stability and sample efficiency with high-dimensional action spaces (like text generation).

• Core Idea: It updates the policy by taking the largest possible improvement step that doesn't destabilize training, using a clipped objective function.
• Key Components in RLHF:
  • Actor: The policy model being fine-tuned.
  • Critic: A separate value network (often initialized from the Reward Model) that estimates the expected reward for a given state.
  • Clipped Surrogate Objective: Prevents overly large policy updates that could collapse performance.
• Role: PPO uses trajectories of (prompt, generated_response, reward) to compute gradients and iteratively improve the policy.

OpenAI's InstructGPT and ChatGPT models are the canonical examples of RLHF deployment. The process involved:

• Collecting human comparisons on model outputs to train a reward model.
• Fine-tuning a GPT-3.5/GPT-4 base model using Proximal Policy Optimization (PPO) against this reward signal.
• Iterative refinement to reduce harmful outputs and improve instruction-following. This implementation proved RLHF's effectiveness for creating helpful, harmless, and honest conversational agents, setting the standard for subsequent large language model alignment.

Anthropic's Claude models utilize a refined RLHF methodology often termed Constitutional AI. This approach incorporates:

• A constitution—a set of principles—to guide AI feedback during training.
• Supervised fine-tuning on AI-generated critiques and revisions based on these principles.
• RL from AI Feedback (RLAIF), where the model itself generates preference data, reducing reliance on extensive human labeling. This implementation emphasizes scalable oversight and the development of models that are steerable, honest, and less likely to produce objectionable content.

DeepMind's Sparrow was a research model designed to be a helpful and harmless dialogue agent. Its RLHF implementation was notable for:

• Training a reward model that prioritized helpfulness, correctness, and harmlessness.
• Incorporating rule-based rewards to penalize violations of specific safety guidelines.
• Enabling the model to cite sources for its claims when retrieving information, adding a layer of verifiability. While not a commercial product, Sparrow provided key insights into making models more truthful and accountable through RLHF.

RLHF has been applied to align code generation models. For systems like Codex (powering GitHub Copilot), the process focuses on programmer preferences:

• Human feedback is gathered on code completions, ranking them based on correctness, efficiency, and readability.
• The reward model learns to predict which code snippets a developer would prefer.
• Fine-tuning via RL steers the model away from generating buggy, insecure, or stylistically poor code. This application demonstrates RLHF's utility beyond natural language, optimizing for functional correctness and developer ergonomics.

RLHF is used to align text-to-image models like OpenAI's DALL-E 3. The implementation involves:

• Generating multiple images for a given prompt and having humans rank them based on prompt adherence, aesthetic quality, and safety.
• Training a multimodal reward model that scores (image, text prompt) pairs.
• Using RL to fine-tune the diffusion model to maximize this reward, improving prompt following and reducing generation of biased or unsafe content. This shows RLHF's extension to multimodal generation, where human feedback is crucial for subjective qualities like style and composition.

While not an end-user product, Direct Preference Optimization (DPO) is a significant algorithmic evolution inspired by RLHF challenges. It reframes the problem by:

• Deriving a closed-form loss from the same human preference data used for reward modeling.
• Directly optimizing the policy language model without training a separate reward model or running complex RL loops like PPO.
• Offering a more stable and computationally efficient alternative that achieves similar alignment results. DPO represents a practical implementation trend, simplifying the RLHF pipeline while maintaining its core objective of preference alignment.

Direct Preference Optimization (DPO) is an algorithm for fine-tuning language models to align with human preferences without training a separate reward model. It reformulates the RLHF objective as a classification loss on human preference data, directly optimizing the policy.

• Mechanism: Uses a closed-form solution derived from the Bradley-Terry model, treating human preferences as the reward signal.
• Advantage: Eliminates the need for proximal policy optimization (PPO) and reward model training, reducing complexity and instability.
• Use Case: A popular, efficient alternative to RLHF for instruction tuning and safety alignment of large language models.

Classifier-Free Guidance (CFG) is a technique for controlling the output of conditional diffusion models by blending the predictions of a conditional and an unconditional model during the sampling process.

• Mechanism: At each denoising step, the model's output is computed as a weighted sum: unconditional_score + guidance_scale * (conditional_score - unconditional_score).
• Purpose: Amplifies the influence of the conditioning signal (e.g., a text prompt) without requiring a separate classifier model.
• Key Parameter: The guidance scale trades off sample quality and diversity against adherence to the condition. High values increase fidelity but can reduce diversity.

Reward Modeling is the process of training a neural network to predict a scalar reward that reflects human preferences, forming the core of the RLHF pipeline. It learns a preference proxy from comparison data.

• Data Collection: Humans rank multiple model outputs for a given prompt, creating pairwise comparison datasets.
• Training Objective: Typically uses the Bradley-Terry model to train the reward model to predict which output a human would prefer.
• Function: The trained reward model provides the reward signal for the subsequent reinforcement learning fine-tuning stage, guiding the base model towards human-aligned outputs.

Proximal Policy Optimization (PPO) is the primary reinforcement learning algorithm used in the fine-tuning stage of RLHF. It optimizes the language model's policy to maximize the expected reward from the reward model while staying close to its original behavior.

• Core Constraint: Uses a clipped objective to prevent large, destabilizing updates to the model parameters.
• RLHF Role: The policy is the language model, the environment is the prompt distribution, and the reward comes from the reward model.
• Challenge: Requires careful tuning to balance reward maximization with the preservation of general language capabilities, often mitigated by adding a KL divergence penalty against the original model.

Constitutional AI is a methodology for training AI systems to be helpful, harmless, and honest using a set of governing principles (a "constitution") rather than solely relying on human feedback for every critique. It often combines RLHF with Reinforcement Learning from AI Feedback (RLAIF).

• Process: The model first generates critiques and revisions of its own outputs based on constitutional principles. These AI-generated preferences are then used to train a reward model.
• Advantage: Scales the generation of preference data and can embed ethical principles directly into the training process.
• Outcome: Aims to create systems that demonstrate chain-of-thought reasoning about their own safety and alignment.

Conditional Generation is the broad class of tasks where a model produces data (text, image, audio) that adheres to specific attributes or instructions provided as an input condition. RLHF is a method for learning the conditioning based on implicit human preferences.

• Conditioning Signals: Can be class labels, text prompts, semantic maps, or other data modalities.
• Architectures: Includes models like Conditional GANs (cGANs), Conditional VAEs (cVAEs), and Conditional Diffusion Models.
• RLHF's Role: RLHF refines a pre-trained conditional generator (e.g., a language model) so its conditional outputs better satisfy a learned reward function based on human preferences.