Alignment Fine-Tuning

RLHF is a multi-stage alignment pipeline. First, a base model is supervised fine-tuned (SFT) on high-quality instruction-response pairs. Next, a separate reward model is trained on human preference data (e.g., choosing between two model outputs). Finally, the SFT model is fine-tuned using a reinforcement learning algorithm like Proximal Policy Optimization (PPO) to maximize the reward predicted by the reward model. This process directly optimizes for outputs humans prefer.

• Key Stages: Supervised Fine-Tuning → Reward Model Training → RL Optimization (PPO).
• Complexity: High, requiring multiple models and a complex training stack.
• Example: Used to train models like OpenAI's InstructGPT and ChatGPT.

Constitutional AI is a methodology developed by Anthropic where a model critiques and revises its own outputs according to a set of written principles (a 'constitution'). The process has two phases:

1. Supervised Constitutional AI: The model generates responses, critiques them based on constitutional principles, and then rewrites them. These (prompt, revised response) pairs are used for SFT.
2. RL from AI Feedback (RLAIF): The model generates multiple responses, uses the constitution to choose a preferred one, and this AI-generated preference data trains a reward model for RL.

• Goal: Reduce reliance on direct human feedback for harmful examples.
• Key Concept: Self-improvement guided by principles.

RLHF is the foundational multi-stage alignment pipeline. It involves:

• Training a reward model on human preference data (e.g., choosing between two model outputs).
• Using that reward model to fine-tune the base language model via a reinforcement learning algorithm like Proximal Policy Optimization (PPO).
• This process directly optimizes the model to generate outputs humans prefer, aligning it with nuanced concepts like helpfulness and harmlessness.

DPO is a more recent, efficient alternative to RLHF that bypasses the need for a separate reward model. Its mechanism includes:

• Deriving a closed-form solution from the same Bradley-Terry preference model used in RLHF.
• Directly optimizing the policy (the language model) using a simple classification loss on preference pairs.
• This results in a stable, computationally lighter training process that often matches or exceeds RLHF performance, making alignment more accessible.

Reward modeling is the critical data-driven component of RLHF. It involves:

• Collecting a dataset of human comparisons (e.g., 'Output A is better than Output B').
• Training a separate neural network (the reward model) to predict a scalar score that reflects these human preferences.
• This model's scores provide the training signal for the RL phase. The quality and bias of the reward model directly dictate the alignment of the final policy.

Constitutional AI is an alignment methodology developed by Anthropic that reduces reliance on extensive human feedback. Its process is:

• Providing the model with a set of written principles or rules (a 'constitution').
• The model uses these principles to critique and revise its own harmful or untruthful responses in a supervised learning phase (RL-CA).
• A final reinforcement learning stage uses AI-generated preferences for training, scaling alignment via self-improvement according to the defined constitution.

Safety fine-tuning is a targeted form of alignment focused on harm prevention. It typically involves:

• Training on a red-teaming dataset containing adversarial prompts designed to elicit harmful, biased, or dangerous outputs.
• Using supervised fine-tuning or reinforcement learning to teach the model to recognize and refuse these requests with a safe, non-compliant response.
• This creates a critical 'refusal layer' and is often a final step before model deployment to ensure basic safety guardrails.

PEFT is not an alignment technique itself but a crucial enabling technology for making alignment feasible. Key methods include:

• LoRA (Low-Rank Adaptation): Injects small, trainable matrices into model layers, leaving original weights frozen.
• Adapter Layers: Inserts small bottleneck modules between transformer layers.
• These methods allow alignment techniques like DPO or SFT to be applied at a fraction of the computational cost, enabling rapid iteration and experimentation on large models.