Reinforcement Learning from Human Feedback (RLHF)

The core mechanism of RLHF involves training a separate reward model on human preference data. Legal annotators are shown two model outputs for the same prompt and asked to select which is more helpful, accurate, and harmless. This creates a dataset of comparative judgments. The reward model learns to predict these human preferences, outputting a scalar score that reflects the quality of a legal response. This score becomes the objective function for the policy optimization phase, effectively distilling complex human legal judgment into a differentiable signal.

RLHF typically uses Proximal Policy Optimization, a reinforcement learning algorithm, to update the language model's weights. The model generates a response, the reward model assigns a score, and PPO updates the policy to maximize that reward. Crucially, PPO includes a KL-divergence penalty that constrains the updated model from deviating too far from its original pre-trained distribution. This prevents the model from 'hacking' the reward model by generating nonsensical but high-scoring text, preserving its general language capabilities while aligning it with human legal standards.

In legal applications, RLHF is critical for training models to refuse harmful requests. Human annotators specifically rank outputs based on harmlessness criteria, teaching the model to decline to provide unauthorized legal advice, generate misleading citations, or assist with illegal activities. An advanced variant, Constitutional AI, replaces some human feedback with a set of written principles (a 'constitution'). The model critiques and revises its own outputs against these rules, enabling scalable alignment with ethical and professional conduct standards without requiring exhaustive human annotation for every edge case.

RLHF directly addresses the problem of legal hallucination—the fabrication of case citations or statutes. During the preference collection phase, human annotators are instructed to penalize outputs containing unverifiable claims. The reward model internalizes this signal, assigning lower scores to fabricated content. Over many training iterations, the policy model learns to associate factual grounding with higher reward, shifting its output distribution toward verifiable statements. This makes RLHF a foundational technique for building legal AI systems with high citation fidelity.

RLHF is not a one-time process. As the model improves, its output distribution shifts, rendering the initial reward model less accurate. This necessitates iterative data collection: the latest policy model generates new outputs, human annotators provide fresh preferences, and a new reward model is trained. This cyclical process—generate, annotate, retrain—is essential for maintaining alignment as the model's capabilities evolve. In legal domains, this ensures the model remains aligned with evolving professional standards and nuanced interpretive frameworks.

Before RLHF begins, the base language model undergoes a critical precursor step: Supervised Fine-Tuning (SFT). Human annotators write high-quality, ideal responses to a diverse set of legal prompts. The model is fine-tuned on this dataset to establish a baseline of helpful behavior. RLHF then refines this SFT model. Starting from an SFT model is essential because directly applying reinforcement learning to a raw pre-trained model is highly unstable. The SFT phase provides a well-initialized policy that already understands the basic format and style of a helpful legal assistant.