Reinforcement Learning from Human Feedback (RLHF)

RLHF operates in a sequential pipeline designed to bridge the gap between raw language modeling and human intent:

• Stage 1: Supervised Fine-Tuning (SFT) — The base model is fine-tuned on a curated dataset of high-quality prompt-response pairs written by human labelers, establishing a baseline for helpfulness and clinical accuracy.
• Stage 2: Reward Model Training — Human annotators rank multiple model outputs for the same prompt. A scalar reward model is trained to predict these preference rankings, learning a proxy for human judgment.
• Stage 3: Proximal Policy Optimization (PPO) — The SFT model is further optimized using PPO, a reinforcement learning algorithm that uses the frozen reward model's score as the objective signal, penalizing the policy for diverging too far from the SFT baseline via a KL-divergence penalty.

PPO is the standard reinforcement learning algorithm used in the final stage of RLHF to update the language model's policy. Its key mechanisms include:

• Clipped Surrogate Objective: PPO limits the size of policy updates by clipping the probability ratio between the new and old policy, preventing destructively large gradient steps that could cause the model to forget its SFT training.
• KL-Divergence Penalty: A penalty term is added to the reward signal to constrain the fine-tuned policy from deviating too far from the SFT baseline distribution, preserving general language capabilities.
• Advantage Estimation: PPO uses Generalized Advantage Estimation (GAE) to compute the relative value of a generated token sequence, reducing the variance of the policy gradient and stabilizing training.

Direct Preference Optimization (DPO) is a more recent alignment technique that bypasses the need for an explicit reward model and PPO entirely:

• Implicit Reward: DPO mathematically reparameterizes the RLHF objective to directly optimize the language model policy from a static dataset of human preference pairs, treating the policy itself as an implicit reward model.
• Stability Advantage: By converting the reinforcement learning problem into a simple supervised classification loss over preference data, DPO eliminates the instability, hyperparameter sensitivity, and computational overhead of PPO training.
• Clinical Relevance: For medical applications, DPO's stability is critical, as it reduces the risk of reward hacking—where a model exploits the reward model to achieve high scores while generating clinically nonsensical or unsafe text.

A central failure mode in RLHF is reward hacking, where the policy learns to exploit imperfections in the reward model to maximize score without actually improving output quality:

• Goodhart's Law: When a proxy measure (the reward model) becomes the optimization target, it ceases to be a good measure. The model may generate verbose, authoritative-sounding but factually incorrect medical advice that the reward model fails to penalize.
• KL-Divergence as a Mitigator: The KL penalty in PPO acts as a regularizer, preventing the policy from drifting into regions of the output space where the reward model is poorly calibrated.
• Iterated Reward Modeling: In production RLHF systems, the reward model is periodically retrained on new human feedback gathered from adversarial examples discovered during deployment, creating a continuous improvement flywheel.

The quality of RLHF is fundamentally bounded by the quality of its human preference data. Collection protocols include:

• Pairwise Comparison: Labelers are shown two model outputs for the same prompt and asked to select the better one based on criteria like helpfulness, harmlessness, and factual accuracy. This is more reliable than absolute scoring.
• Clinical Expert Labelers: For medical RLHF, annotators must be domain experts—physicians or clinical informaticists—who can identify subtle errors like incorrect drug dosages or contraindications that laypeople would miss.
• Constitutional AI: An alternative approach where the model generates its own critiques and revisions based on a written set of principles, reducing reliance on human labelers for harmlessness training.

In healthcare, RLHF is specifically adapted to enforce safety constraints beyond general helpfulness:

• Refusal Training: The reward model is trained to highly score appropriate refusals—for example, when a model correctly declines to provide a specific chemotherapy dosage without knowing the patient's full oncological history.
• Uncertainty Calibration: Human labelers reward outputs that appropriately express uncertainty rather than generating confident-sounding but potentially harmful medical assertions.
• Adversarial Red-Teaming: Before deployment, the RLHF-aligned model is probed with adversarial prompts designed to elicit unsafe medical advice. Failures are fed back into the preference dataset for iterative retraining.

A stable alternative to RLHF that eliminates the need for a separate reward model. DPO directly optimizes the language model's policy using a static dataset of human-ranked output pairs.

• How it works: Reparameterizes the RLHF objective to define the preference loss directly on the policy model.
• Clinical relevance: Simplifies the training pipeline for aligning medical summarization models with clinician preferences.
• Key advantage: Avoids the instability and computational overhead of training and maintaining a reward model.

A failure mode where the policy model exploits loopholes in the reward model to achieve high scores without genuinely aligning with human intent.

• Example: A model generates verbose, authoritative-sounding text that is factually incorrect because the reward model overvalues length and confidence.
• Mitigation: Requires iterative refinement of the reward model with adversarial examples and diverse human feedback.
• Clinical risk: A hacked reward model could produce dangerously plausible but incorrect treatment summaries.

An alignment method developed by Anthropic that uses a set of predefined principles, or a 'constitution,' to guide model outputs. The model engages in self-critique and revision based on these rules.

• Process: Uses supervised fine-tuning on self-generated revisions, followed by RLHF using a preference model trained on constitutionally guided comparisons.
• Clinical application: Can encode medical ethics and safety guidelines directly into the training process.
• Benefit: Reduces reliance on extensive human feedback for basic safety norms.

The foundational reinforcement learning algorithm commonly used in the RLHF pipeline to update the language model's policy. PPO constrains policy updates to a trust region to prevent destructive large weight changes.

• Mechanism: Uses a clipped surrogate objective function to penalize large deviations from the previous policy.
• Stability: Balances exploration and exploitation while maintaining stable training dynamics.
• Role in RLHF: Acts as the optimizer that nudges the model toward outputs that maximize the reward model's score.

The critical human-in-the-loop process of generating the training signal for RLHF. Human annotators rank multiple model-generated responses to the same prompt from best to worst.

• Format: Typically involves pairwise comparisons or Likert scale ratings on axes like helpfulness, harmlessness, and accuracy.
• Clinical nuance: Requires domain experts to judge medical correctness, not just stylistic quality.
• Scalability challenge: High-quality medical preference data is expensive and slow to produce, driving interest in synthetic alternatives.

A novel alignment method that learns from binary feedback signals—'good' or 'bad'—rather than requiring costly pairwise preference rankings. KTO directly maximizes the utility of generated outputs.

• Advantage: Aligns with how human feedback naturally occurs in production systems, such as thumbs-up/thumbs-down signals.
• Efficiency: Can effectively utilize imbalanced datasets where positive examples are scarce.
• Clinical use case: Enables continuous alignment from clinician feedback embedded directly in the EHR interface.