Safety Fine-Tuning

RLHF is a three-stage alignment pipeline that fine-tunes a base language model using a reward model trained on human preference data.

• Stage 1: Supervised Fine-Tuning (SFT) – The base model is first fine-tuned on high-quality, human-written demonstrations of desired behavior.
• Stage 2: Reward Modeling – A separate model is trained to predict human preferences, scoring responses as more or less desirable based on safety and helpfulness.
• Stage 3: Reinforcement Learning – The SFT model is optimized via algorithms like Proximal Policy Optimization (PPO) to maximize the score from the reward model, shaping its outputs toward human-aligned values.

DPO is a stable, efficient alternative to RLHF that directly optimizes a language model policy using a dataset of preferred and dispreferred responses, eliminating the need to train and maintain a separate reward model.

• Mechanism: It treats the language model itself as an implicit reward function, using a closed-form solution derived from the reward modeling objective.
• Advantages: DPO is simpler to implement, more stable during training, and computationally cheaper than the full RLHF pipeline, while achieving comparable alignment performance.
• Use Case: Ideal for rapid iteration on safety fine-tuning where reward model training is a bottleneck.

Constitutional AI is a framework where a model is trained to critique and revise its own outputs against a set of written principles (a 'constitution'). Reinforcement Learning from AI Feedback (RLAIF) scales this process.

• Self-Critique Loop: The model generates a response, then uses the constitution to produce a critique of that response, and finally revises it to address the critique.
• AI-Generated Preferences: These self-critiqued revisions create a dataset of AI-preferred outputs, which is then used to train a preference model, enabling RLAIF as a scalable alternative to human feedback.
• Benefit: Enables scalable, principle-driven alignment without continuous human oversight for every preference label.

This technique involves curating and training on datasets explicitly designed to teach safe refusal, ethical reasoning, and harm avoidance.

• Dataset Composition: Includes examples of harmful queries paired with refusal responses that explain the safety policy, as well as benign queries paired with helpful, harmless answers.
• Red-Teaming Data: Incorporates outputs from automated red-teaming where adversarial models generate attack prompts, and the target model's successful refusals are used as training data.
• Objective: Directly teaches the model the desired input-output mapping for safety-critical edge cases, building a foundational understanding of boundaries before applying preference optimization.

These are inference-time techniques that constrain model outputs to enforce safety, acting as a final protective layer after fine-tuning.

• Constrained Decoding: Restricts the model's token-by-token generation to a subset of permissible outputs defined by a grammar or allowed list, preventing the generation of banned phrases or formats.
• Steering Vectors: Small vectors are added to the model's internal activations during inference to amplify or suppress certain concepts (e.g., reducing toxicity vectors).
• Safety Classifiers & Filters: External safety classifier models or rule-based filters scan the model's final output, blocking or rewriting text that violates policies before it reaches the user.

Advanced techniques that aim to directly edit a model's neural weights to remove specific dangerous knowledge or behavioral tendencies without retraining the entire network.

• Mechanisms: Methods like Rank-One Model Editing (ROME) or Knowledge Neurons identification locate and modify parameters associated with a harmful concept (e.g., bomb-making).
• Goal: To 'erase' the model's ability to generate content related to the targeted harmful concept while preserving its general capabilities and performance on other tasks.
• Challenge: Requires precise localization to avoid catastrophic side effects on unrelated knowledge, making it an active area of safety research.

Harmful concept erasure is a targeted fine-tuning or model editing technique aimed at removing or neutralizing specific dangerous knowledge or behavioral tendencies from a model's weights.

• Objective: 'Unlearn' the ability to generate specific unsafe content (e.g., detailed illegal instructions) without degrading general performance.
• Techniques: Can involve contrastive fine-tuning on pairs of safe/unsafe examples or more advanced parameter editing methods.

A safety classifier is a separate machine learning model, often deployed in tandem with a fine-tuned LLM, that analyzes text to detect specific categories of harmful content.

• Function: Acts as a runtime filter or a reward signal during training. It classifies inputs or outputs for toxicity, violence, unethical advice, etc.
• Deployment: Used for output verification and as a critical component in refusal mechanisms and governance hooks.

A refusal mechanism is a programmed behavior, often instilled via safety fine-tuning, where an AI system declines to generate a response when a query violates its safety policies.

• Key Feature: Includes an explainable refusal that justifies the decision by citing the violated principle.
• Implementation: Trained by fine-tuning on datasets containing examples of harmful queries paired with polite, principled refusals.