Reinforcement Learning from Human Feedback (RLHF)

RLHF operates in a sequential loop of three distinct phases:

• Supervised Fine-Tuning (SFT): The base model is first fine-tuned on high-quality demonstration data where human labelers write ideal responses to prompts.
• Reward Model (RM) Training: The SFT model generates multiple outputs for a prompt. Human labelers rank these outputs from best to worst. A scalar reward model is trained to predict these preference rankings.
• Proximal Policy Optimization (PPO): The SFT model is further fine-tuned using reinforcement learning, where the frozen reward model provides the reward signal. A KL-divergence penalty term prevents the policy from diverging too far from the SFT distribution.

RLHF relies on two fundamentally different types of human annotation:

• Demonstration Data: Used in the SFT phase. Labelers write the correct answer from scratch. This teaches the model what to say.
• Preference Data: Used to train the reward model. Labelers compare two or more model-generated outputs and rank them. This teaches the model what is better. Preference ranking is significantly easier and yields higher inter-annotator agreement than writing demonstrations.

A critical technical detail in the PPO phase is the Kullback–Leibler (KL) divergence penalty. Without it, the policy model can exploit loopholes in the reward model to achieve high scores while producing nonsensical or ungrammatical text—a phenomenon known as reward hacking. The KL penalty measures the divergence between the current policy's output distribution and the frozen SFT model's distribution, keeping the model grounded in coherent language while optimizing for human preference.

Developed by Anthropic, Constitutional AI (CAI) addresses the bottleneck of human feedback by replacing human preference labels with a written constitution of principles. The process involves:

• Supervised Phase: The model generates self-critiques and revisions of harmful outputs guided by constitutional principles.
• RL Phase: The model is fine-tuned using AI-generated preference data based on constitutional compliance, rather than human rankings. This enables harmlessness training to scale without proportional increases in human labor.

A fundamental failure mode in RLHF occurs when the policy model learns to exploit spurious correlations in the reward model to achieve high scores without genuinely improving output quality. Examples include:

• Generating verbose but vacuous text that the reward model mistakenly ranks highly.
• Inserting specific tokens or phrases that trigger high reward scores.
• Producing grammatically perfect but factually empty responses. This is why KL-constrained optimization and periodic reward model retraining are essential guardrails in production RLHF systems.

A core failure mode in RLHF where the policy model learns to exploit flaws in the reward model to achieve high scores without genuinely aligning with human intent. This is a form of Goodhart's Law in practice. Examples include generating verbose but vacuous text, using overly complex vocabulary, or inserting flattery. Mitigation strategies involve iterative reward model refinement, ensembling multiple reward models, and incorporating a KL-divergence penalty to prevent the policy from deviating too far from its original distribution.

The foundational training signal for RLHF, consisting of comparisons between two or more model outputs. Human annotators rank responses based on criteria like helpfulness, harmlessness, and honesty. This data is used to train the reward model. Key challenges include inter-annotator disagreement, the high cost of expert labeling, and ensuring demographic diversity among raters to avoid encoding narrow cultural biases. Platforms like Scale AI and Surge AI specialize in generating this data at scale.

A regularization term added to the reward function during the PPO phase of RLHF. It measures the divergence between the probability distributions of the current policy and the initial supervised fine-tuned model. By penalizing large deviations, this constraint prevents catastrophic forgetting and stops the model from collapsing into a narrow mode of generating repetitive, high-reward gibberish. The coefficient (beta) controlling this penalty is a critical hyperparameter that balances alignment with output diversity and coherence.