Reinforcement Learning from Human Feedback (RLHF)

The foundational step where a pre-trained base model is fine-tuned on a high-quality dataset of human-written demonstrations for the target task. This teaches the model the desired output format and stylistic conventions before preference optimization begins.

• Input: A curated set of (prompt, ideal_response) pairs.
• Purpose: Establishes a strong, well-behaved policy baseline, preventing the subsequent RL stage from diverging too far from coherent language.
• Example: Training on a dataset of legal summarization examples where human experts have written the perfect, citation-backed summary.

The human-in-the-loop process of generating a dataset that captures subjective human judgment. For a single prompt, the SFT model generates multiple candidate outputs, and human labelers rank them from best to worst.

• Key Metric: Inter-annotator agreement, which measures the consistency of human preferences.
• Format: Typically structured as triplets (prompt, chosen_response, rejected_response).
• Legal Context: Labelers might rank summaries based on citation integrity and factual accuracy over stylistic flair.

A separate model, often initialized from the SFT checkpoint, trained to predict the human preference score. It learns to assign a scalar reward that is higher for the 'chosen' response and lower for the 'rejected' one.

• Architecture: The final layer is replaced with a regression head that outputs a single scalar value.
• Loss Function: Typically uses a pairwise ranking loss to maximize the margin between the preferred and dispreferred output scores.
• Function: Acts as a proxy for human judgment, providing a differentiable reward signal for the policy.

The reinforcement learning algorithm most commonly used to update the language model policy using the frozen reward model's signal. The policy generates a response, the RM scores it, and PPO updates the policy to maximize the reward.

• KL-Divergence Penalty: A critical constraint that prevents the new policy from straying too far from the SFT baseline, avoiding reward hacking and preserving language coherence.
• Objective: Maximize E[reward - β * KL(π_new || π_SFT)].
• Outcome: A policy that is optimized to produce outputs humans judge as more helpful, truthful, and harmless.

A more recent and stable alternative to the full PPO pipeline. DPO bypasses the need to train a separate reward model by directly optimizing the language model on the preference data using a binary cross-entropy loss.

• Mechanism: Reparameterizes the RL objective to directly increase the log-likelihood of preferred responses relative to dispreferred ones.
• Advantage: Eliminates the instability and computational overhead of training and running a separate reward model during the RL loop.
• Stability: Less prone to reward hacking compared to online PPO methods.

In production-grade systems, RLHF is not a single pass but a continuous cycle. The newly optimized policy is deployed, gathers new human feedback on its failures, and this fresh preference data is used to train a superior reward model for the next round of optimization.

• Process: Deploy -> Collect Feedback -> Retrain RM -> Retrain Policy.
• Benefit: Creates a virtuous cycle that continuously aligns the model with evolving human expectations and edge cases.
• Application: Essential for legal AI to constantly improve on rare, complex multi-document reasoning tasks.

A failure mode in RLHF where the language model learns to exploit loopholes in the reward model to achieve high scores without genuinely improving output quality. The model optimizes for what the reward model measures rather than what humans value.

• Common in legal AI when reward models over-prioritize fluency over factual accuracy
• Can produce confident-sounding but hallucinated legal citations that 'look correct' to the reward model
• Mitigated through iterative reward model refinement and adversarial training
• Requires careful balance between helpfulness, harmlessness, and honesty reward components

The foundational process of gathering human comparison judgments between pairs of model outputs to train the reward model in RLHF. The quality and consistency of this data directly determines alignment success.

• Human annotators rank two model responses based on criteria like truthfulness, helpfulness, and harmlessness
• Legal preference data requires domain-expert annotators who can identify subtle factual errors
• Inter-annotator agreement is critical; legal experts often disagree on nuanced interpretations
• Synthetic preference data from Constitutional AI can augment scarce expert human judgments

A preference optimization method that learns from unpaired preference data—individual examples labeled as 'good' or 'bad' rather than requiring pairwise comparisons. Named after the psychologists who studied human decision-making biases.

• Dramatically reduces data collection costs by eliminating the need for paired comparisons
• Particularly useful in legal domains where creating balanced comparison pairs is difficult
• Aligns models based on whether outputs are desirable or undesirable in absolute terms
• Enables leveraging existing legal document quality ratings and expert feedback signals