Safety Alignment

RLHF is a multi-stage alignment pipeline. First, a reward model is trained on a dataset of human preferences, where labelers rank multiple model outputs. This reward model learns to predict a scalar score representing human preference. The main language model is then fine-tuned using Proximal Policy Optimization (PPO) to maximize the reward predicted by this model, thereby aligning its outputs with human values. This process directly optimizes for desirable behavior as defined by human raters.

• Core Stages: Supervised Fine-Tuning (SFT) → Reward Model Training → RL Fine-Tuning (PPO).
• Key Component: The reward model acts as a differentiable proxy for human judgment.
• Challenge: Requires extensive, high-quality human preference data and is computationally intensive.

DPO is an alternative to RLHF that eliminates the need for a separate reward model and the complex reinforcement learning loop. It derives a closed-form solution for the optimal policy under the Bradley-Terry preference model. The algorithm directly optimizes the language model using a simple classification loss on pairs of preferred and dispreferred outputs.

• Mechanism: Treats the language model itself as an implicit reward function.
• Advantages: More stable and computationally efficient than RLHF, as it avoids unstable PPO training.
• Use Case: Widely adopted for its simplicity and effectiveness in aligning models like Llama 2 and Mistral.

Constitutional AI (CAI) is a methodology where an AI model critiques and revises its own outputs according to a predefined set of principles or a 'constitution'. Reinforcement Learning from AI Feedback (RLAIF) extends this by using a constitutionally-guided AI to generate preference data for training a reward model, which then fine-tunes the main model via RL.

• Self-Critique: The model generates a response, then critiques it against the constitution, and finally revises it.
• Scalability: Reduces reliance on extensive human feedback for every preference pair.
• Principle-Based: Alignment is driven by explicit, written principles rather than implicit preferences.

This technique proactively strengthens model safety by exposing it to adversarial examples during training. Red teaming involves human or automated attempts to generate prompts that 'jailbreak' the model or elicit harmful outputs. These adversarial examples are then incorporated into the training data.

• Process: Generate harmful prompts → Use model to produce (potentially unsafe) completions → Fine-tune the model to refuse or handle these prompts safely.
• Goal: Improves robustness by teaching the model to recognize and resist manipulation.
• Iterative: Often performed in cycles of attack generation and model patching.

This is the foundational step for many alignment pipelines. The model is fine-tuned on a high-quality safety dataset containing demonstrations of desired behavior. This dataset includes:

• Safe Demonstrations: Examples of helpful, harmless, and honest responses.
• Refusal Training: Explicit examples of the model appropriately declining harmful or out-of-scope requests.
• Toxicity Mitigation: Paired examples of toxic inputs and non-toxic, corrected responses.
• Role: Provides a strong behavioral prior before more advanced preference optimization techniques are applied.

This category includes algorithms that optimize for preferences without pairwise ranking data. Kahneman-Tversky Optimization (KTO) trains using binary feedback (thumbs up/down) on single outputs, leveraging loss aversion from prospect theory—penalizing undesirable outputs more heavily than rewarding desirable ones.

• KTO Mechanism: Uses a sigmoid loss on the reward difference between positive and negative examples.
• Identity Preference Optimization (IPO): Adds a regularization term to prevent overfitting to the preference data, improving generalization.
• Advantage: Can utilize simpler, more abundant binary feedback signals instead of curated pairwise comparisons.