Reinforcement Learning from Human Feedback (RLHF)

Instead of engineering a scalar reward function by hand, RLHF trains a reward model on a dataset of human comparisons. For a given prompt, the model generates multiple responses, and human labelers rank them. The reward model learns to predict these rankings, outputting a score that serves as a proxy for human judgment. This approach captures complex, subjective qualities like helpfulness and harmlessness that are impossible to codify in explicit rules.

The final stage of RLHF typically uses Proximal Policy Optimization, a policy gradient algorithm, to fine-tune the language model. The trained reward model provides the reward signal, while a KL divergence penalty term is added to the loss function. This penalty prevents the policy from diverging too far from the supervised fine-tuned model, mitigating reward hacking—where the model exploits loopholes in the reward model to achieve high scores without genuinely aligning with human intent.

RLHF is not a single algorithm but a sequential pipeline with three distinct phases:

• Stage 1: Supervised Fine-Tuning (SFT). A pre-trained language model is fine-tuned on high-quality demonstration data where humans write ideal responses.
• Stage 2: Reward Model Training. The SFT model generates response pairs, humans rank them, and a reward model is trained to predict these preference scores.
• Stage 3: Policy Optimization. The SFT model is further fine-tuned using PPO, with the frozen reward model providing the learning signal.

A more recent alternative to the classic three-stage RLHF pipeline is Direct Preference Optimization (DPO). DPO eliminates the need for a separate reward model entirely by reparameterizing the reward function in terms of the optimal policy. The language model is fine-tuned directly on the human preference dataset using a binary cross-entropy loss. This simplifies the pipeline, removes the complexity of PPO tuning, and is more stable, though it lacks the flexibility of an explicit reward model for iterative refinement.

A key limitation of RLHF is the cost and scalability of human feedback. Reinforcement Learning from AI Feedback (RLAIF) addresses this by replacing human labelers with a separate, powerful language model. In Constitutional AI, the model generates self-critiques and revisions based on a predefined set of principles (a 'constitution'). The AI-generated preference data is then used to train a reward model or directly optimize the policy via DPO, enabling scalable alignment with reduced human oversight.

The RLHF process often incurs an alignment tax—a measurable degradation in certain capabilities, such as reasoning or factual recall, as the model becomes more aligned with human preferences. This creates a Pareto frontier between helpfulness and harmlessness. Excessive optimization against the reward model can also lead to overoptimization, where the policy achieves high reward scores but produces sycophantic, repetitive, or stylistically narrow outputs that diverge from true human intent.

The formal mathematical framework for modeling sequential decision-making in stochastic environments. An MDP is defined by a tuple (S, A, P, R, γ):

• S: A set of states representing the environment
• A: A set of actions the agent can take
• P(s'|s, a): The transition probability of moving to state s' after taking action a in state s
• R(s, a): The immediate reward received
• γ (gamma): The discount factor (0 ≤ γ ≤ 1) that balances immediate vs. future rewards

In RLHF, the language model's generation process is formalized as an MDP where states are partial text sequences and actions are token selections. The Markov property—that future states depend only on the present state, not the history—is a critical assumption.

The fundamental dilemma in sequential decision-making where an agent must balance two competing objectives:

• Exploration: Trying new or uncertain actions to gather more information about the environment and potentially discover better strategies
• Exploitation: Choosing the best-known action based on current knowledge to maximize immediate reward

In RLHF, this manifests during policy training. The model must exploit known high-reward responses (those aligned with human preferences) while exploring the vast space of possible outputs to avoid converging on suboptimal, repetitive, or sycophantic behaviors. Techniques like entropy bonuses and epsilon-greedy sampling are used to maintain sufficient exploration.

A class of reinforcement learning algorithms that directly optimize the policy function π(a|s)—the probability of taking action a in state s—by estimating the gradient of expected cumulative reward with respect to policy parameters.

Key algorithms include:

• REINFORCE: The simplest Monte Carlo policy gradient method
• Proximal Policy Optimization (PPO): The dominant algorithm used in RLHF implementations, which constrains policy updates to a trust region to prevent destructive large updates
• Advantage Actor-Critic (A2C/A3C): Combines policy gradients with a value function baseline to reduce variance

PPO is the workhorse of modern RLHF because it provides stable, reliable convergence when fine-tuning large language models against a learned reward model.

The process of training a separate model to predict human preferences, which then serves as the reward function for policy optimization. This is the 'HF' in RLHF.

The standard pipeline:

1. Collect human pairwise comparisons between model outputs (e.g., 'Response A is better than Response B')
2. Train a reward model r_θ(x, y) that scores outputs y given prompts x
3. Use the Bradley-Terry model to convert comparisons into a loss function: P(A > B) = exp(r(A)) / (exp(r(A)) + exp(r(B)))

The reward model is typically initialized from the same base language model, with the final layer replaced by a scalar regression head. Its accuracy directly determines RLHF success.

A regularization term added to the RLHF objective to prevent the policy from diverging too far from the original supervised fine-tuned model. The full objective becomes:

max_π E[r(x,y)] - β * KL(π || π_ref)

Where:

• r(x,y): The reward model's score
• KL(π || π_ref): Kullback-Leibler divergence between current and reference policy
• β: A hyperparameter controlling the strength of the constraint

This prevents reward hacking—where the policy exploits loopholes in the reward model to achieve high scores while producing nonsensical or degenerate outputs. It also preserves general capabilities learned during pre-training.

An alternative alignment approach developed by Anthropic that reduces reliance on human feedback by using a constitution—a set of written principles—to guide model behavior.

The two-phase process:

• Supervised Phase: The model generates responses, critiques them against constitutional principles, and revises them. These revisions form training data for fine-tuning.
• RL Phase: Instead of a human-trained reward model, the model uses AI feedback based on constitutional principles to score responses during reinforcement learning.

This addresses the scalability bottleneck of human annotation while maintaining explicit, auditable behavioral guidelines. It is often combined with RLHF in a hybrid approach.