Value Alignment

RLHF is the foundational technique for aligning large language models. It involves three key stages:

• Supervised Fine-Tuning (SFT): A base model is initially fine-tuned on high-quality, human-written demonstrations of desired behavior.
• Reward Model Training: A separate model is trained to predict human preferences by learning from datasets where humans rank multiple model outputs.
• Reinforcement Learning Fine-Tuning: The main policy model is optimized against the learned reward model using algorithms like Proximal Policy Optimization (PPO), encouraging outputs that score highly according to human preferences. This process directly encodes nuanced human judgments about helpfulness, harmlessness, and honesty into the model's parameters.

Constitutional AI is a framework where an AI's behavior is governed by a predefined set of principles or a 'constitution'. The core innovation is Reinforcement Learning from AI Feedback (RLAIF):

• Self-Critique and Revision: The model critiques its own responses against the constitutional principles and generates revisions.
• AI-Generated Preferences: These revised responses are used to train a preference model, replacing human labelers in the RLHF loop.
• Scalable Principle Enforcement: This creates a scalable method for alignment, as the 'constitution' provides a clear, auditable source of truth. Principles might include "Choose the response that is most supportive and encouraging" or "Prioritize privacy and confidentiality."

DPO is a more stable and computationally efficient alternative to RLHF that bypasses the need to train a separate reward model.

• Mathematical Reformulation: DPO reframes the RLHF objective so that the optimal policy can be derived directly from the preference data.
• Direct Policy Optimization: The language model itself is fine-tuned directly on datasets containing pairs of preferred and dispreferred responses, eliminating the unstable RL loop.
• Key Advantages: It reduces complexity, training cost, and hyperparameter sensitivity while achieving comparable or superior alignment performance. It is particularly effective for further refining models that have undergone initial RLHF.

This approach involves adding specialized, defense-in-depth layers to enforce alignment during and after model training:

• Safety Fine-Tuning: The model undergoes additional training on datasets specifically designed to teach refusal mechanisms for harmful requests and to improve performance on sensitive topics.
• Harm Classification: Dedicated safety classifier models (e.g., for toxicity, violence, illegal advice) scan inputs and outputs to trigger interventions.
• Constrained Decoding: Inference-time techniques that restrict the model's vocabulary or apply logit biases to prevent the generation of certain tokens or phrases.
• Governance Hooks: Middleware that applies input/output validation, audit trail generation, and policy-as-code rules before the user receives a response.

These are inference-time and parameter-level techniques for precise behavioral control:

• Steering Vectors: Adding specific direction vectors to a model's internal activations can steer its outputs toward desired attributes (e.g., formality, creativity) or away from harmful concepts.
• Activation Engineering: Manually or automatically patching neural activations in real-time to correct for biases or unsafe reasoning.
• Harmful Concept Erasure: Advanced fine-tuning methods, such as Rank-One Model Editing (ROME), that aim to directly edit a model's weights to 'forget' or neutralize specific dangerous knowledge or behavioral patterns without retraining the entire network.

Proactive testing and hardening are critical for robust alignment. This involves systematically probing for failures:

• Automated Red-Teaming: Using AI models themselves to generate vast numbers of adversarial jailbreak prompts and edge-case queries designed to bypass safety filters.
• Adversarial Training: The failed examples from red-teaming are incorporated into the model's training data, teaching it to recognize and resist similar attacks in the future.
• Jailbreak Detection: Deploying dedicated models or heuristics to identify and block adversarial prompt patterns before they reach the main model. This cycle of attack and defense is essential for achieving adversarial robustness and closing safety loopholes before deployment.