Constitutional AI

A structured set of normative principles that define acceptable and unacceptable outputs. These rules are written in natural language and cover topics like toxicity, bias, illegal content, and ethical guidelines.

• Principle examples: 'Do not encourage violence,' 'Respect user privacy,' 'Choose the response that is most helpful and least harmful.'
• Origin: Anthropic pioneered this approach to reduce reliance on human labelers for RLHF.
• Transparency: The constitution is auditable and can be publicly shared, unlike opaque human preference data.

The model generates an initial response to a potentially harmful prompt, then critiques and revises its own output based on the constitution.

• Phase 1: Model produces a response.
• Phase 2: Model evaluates the response against each constitutional principle.
• Phase 3: Model rewrites the response to eliminate identified violations.
• This creates a self-supervised dataset of (harmful prompt, revised safe response) pairs for fine-tuning.

Instead of training a reward model on human preferences, Constitutional AI trains a preference model using AI-generated feedback.

• The model evaluates pairs of responses based on constitutional principles.
• This AI feedback signal replaces human labelers in the reinforcement learning loop.
• Scalability: Eliminates the bottleneck of human annotation for harmlessness.
• Consistency: AI evaluations are deterministic and exhaustively documented, unlike subjective human judgments.

Constitutional AI operates in two distinct phases:

Stage 1: Supervised Fine-Tuning

• Model generates revised responses via self-critique.
• Fine-tune on the resulting (prompt, safe response) dataset.

Stage 2: Reinforcement Learning

• Train a preference model using AI feedback on constitutional compliance.
• Apply PPO (Proximal Policy Optimization) to align the model with the AI preference model.
• This dual approach ensures both immediate safety and long-term alignment.

A key advantage over standard RLHF is calibrated refusal boundaries. Because the constitution explicitly defines what constitutes harm, the model learns to distinguish between genuinely harmful requests and benign queries that merely contain sensitive keywords.

• RLHF problem: Human labelers often over-correct, causing models to refuse safe requests.
• CAI solution: The constitution provides nuanced, context-aware guidance.
• Result: Higher helpfulness scores without compromising safety.

The constitution can explicitly encode anti-discrimination and fairness principles to mitigate social biases.

• Principles like 'Do not stereotype based on race, gender, or religion' directly guide the self-critique process.
• Unlike human feedback, which can inadvertently reinforce societal biases, a well-crafted constitution provides a consistent ethical baseline.
• The constitution can be iteratively updated to address newly discovered failure modes without retraining the entire RLHF pipeline.

The fundamental reasoning loop that powers Constitutional AI. In the supervised phase, the model generates a response to a harmful prompt, then critiques its own output against each constitutional principle, and finally revises the response to remove violations. This chain-of-thought revision process teaches the model to internalize the constitution as an editing heuristic. Key steps include:

• Critique: Identify specific ways the response violates a principle
• Revision: Rewrite the response to address all identified issues
• Finalization: Output only the revised, compliant response

The adversarial evaluation methodology used to stress-test Constitutional AI systems. Automated red teaming generates diverse attack vectors—including jailbreaks, prompt injections, and edge-case scenarios—to probe for constitutional violations. Findings from red teaming exercises are fed back into the constitution itself, adding new principles to cover discovered vulnerabilities. This creates a flywheel effect: red teaming identifies gaps, the constitution is amended, and the model is retrained to close those gaps, continuously hardening the alignment surface.

A lightweight guard model that operates as a runtime complement to Constitutional AI. While Constitutional AI bakes safety into the model weights during training, a safety classifier provides a defense-in-depth layer at inference time. It scores prompts and outputs for policy violations (toxicity, violence, PII leakage) and can trigger refusal strings or circuit breakers before content reaches the user. Common architectures include fine-tuned DeBERTa-v3 variants that run in under 10ms, providing low-latency protection without modifying the base model.

An emerging technique that directly manipulates a model's internal activations to control behavior, bypassing prompt-based constitutional oversight. Researchers identify safety vectors—directions in the latent space that correspond to honesty or harmlessness—and add them to the forward pass during inference. This provides a hardware-level guarantee that the model operates in a safe regime, complementing Constitutional AI's training-time alignment with runtime activation steering. It is particularly effective against adversarial suffixes that attempt to jailbreak the model.