Constitutional AI

The model is given a constitution—a list of natural language principles—and trained to critique its own responses against these rules. During the supervised phase, the model generates a response, then revises it based on constitutional feedback, creating a dataset of harmless outputs without human annotators evaluating harmfulness.

• Principles cover topics like avoiding toxicity, respecting privacy, and not providing dangerous information
• The model learns to identify violations and rewrite responses accordingly
• Eliminates the bottleneck of human harmfulness labeling

Constitutional AI uses a two-stage pipeline distinct from standard RLHF. In the first phase, the model performs supervised self-revision, generating initial responses and then critiquing and rewriting them according to constitutional principles. In the second phase, the model is fine-tuned using Reinforcement Learning from AI Feedback (RLAIF), where a separate model evaluates harmlessness based on the same constitution.

• Phase 1: Supervised learning on self-revised outputs
• Phase 2: RLAIF replaces human preference judgments with AI-generated evaluations
• Both phases rely entirely on the written constitution rather than human harmlessness ratings

Instead of collecting human preference data to train a reward model, Constitutional AI uses AI-generated feedback. The model is shown pairs of responses and asked to select which one better adheres to the constitution. This preference data trains a reward model that scores outputs for harmlessness during reinforcement learning.

• Removes human annotators from the harmlessness evaluation loop entirely
• Scales more efficiently than human feedback collection
• Maintains consistency by applying the same constitutional principles at every stage

The constitution is a publicly auditable document that explicitly defines the values guiding the model's behavior. Unlike RLHF, where human preferences are implicit and can vary across annotators, Constitutional AI makes the training objectives transparent and inspectable. Organizations can customize the constitution to encode domain-specific ethical guidelines.

• Principles are explicit, version-controlled, and open to scrutiny
• Enables domain-specific constitutional customization for enterprise deployments
• Provides a clear audit trail for AI governance and compliance requirements

Standard RLHF requires human annotators to read and rate potentially disturbing or toxic model outputs to train harmlessness reward models. Constitutional AI eliminates this requirement for harmlessness training by using the model's own critique capabilities. Human feedback is still used for helpfulness, but the harmlessness axis is fully automated.

• Protects human annotators from exposure to harmful content
• Dramatically reduces the cost and time of safety training
• Enables rapid iteration on safety principles without re-annotation

Because the constitution is applied programmatically rather than through human judgment, Constitutional AI scales to larger models and more complex behaviors without proportional increases in annotation cost. As models become more capable, the same constitutional principles can guide increasingly sophisticated self-critique, creating a scalable alignment framework.

• Principles remain valid as model capabilities grow
• Self-critique quality improves with model capability
• Enables alignment research to proceed without being bottlenecked by human evaluation bandwidth

The precursor and close relative to Constitutional AI. RLHF trains a reward model on human preference comparisons, then uses Proximal Policy Optimization (PPO) to fine-tune the language model. Unlike Constitutional AI, RLHF relies on human annotators to evaluate outputs for harmlessness and helpfulness, creating a bottleneck of human judgment. Constitutional AI replaces this human feedback loop with AI-generated self-critiques guided by a written constitution.

A stable alternative to both RLHF and Constitutional AI's training pipeline. DPO reparameterizes the reward function directly in terms of the optimal policy, eliminating the need to train a separate reward model. This avoids the reward hacking and instability common in PPO-based methods. While Constitutional AI uses a constitution to generate preference data, DPO provides a simpler mathematical framework for learning from that data.

The production deployment counterpart to Constitutional AI's training-time principles. Guardrail injection embeds non-negotiable safety rules directly into the system prompt or generation logic. While Constitutional AI bakes principles into model weights during training, guardrails enforce constraints at inference time. Common implementations include:

A deterministic complement to Constitutional AI's probabilistic self-critique. Constrained decoding forces every generated token to conform to a predefined formal grammar or schema, guaranteeing structural validity. Where Constitutional AI asks the model to revise harmful content, constrained decoding prevents malformed outputs from being generated at all. Used heavily in JSON mode and API response formatting.

A parallel safety objective that Constitutional AI indirectly supports. Hallucination mitigation encompasses techniques to reduce factually incorrect outputs, including retrieval-augmented generation (RAG), factuality scoring, and grounding attribution. Constitutional AI's self-critique phase can catch factual errors, but its primary focus is harmlessness and ethical alignment, not factual accuracy. The two disciplines are complementary layers in a production safety stack.

The mechanism of linking each factual claim to a verifiable source, which Constitutional AI can leverage during its revision phase. Grounding attribution requires the model to cite specific documents or data provenance for its statements. When combined with Constitutional AI, the self-critique step can check not only for harmlessness but also for evidentiary support, rejecting claims that lack proper attribution to the provided context.