Constitutional AI (CAI)

A constitution is a curated list of natural language principles defining ideal behavior. During the supervised phase, the model generates responses to harmful prompts, then critiques and revises its own output based on these principles. This creates a self-supervised dataset for fine-tuning, eliminating the need for humans to read disturbing content. The constitution typically draws from sources like the UN Declaration of Human Rights, Apple's terms of service, and custom safety guidelines.

Traditional RLHF requires humans to compare model outputs and express preferences, a bottleneck that exposes workers to harmful content. RLAIF (Reinforcement Learning from AI Feedback) replaces the human judge with a language model prompted with the constitution. The AI feedback model evaluates which of two responses better adheres to the principles, producing a preference label. This makes the harmlessness training process fully automated and scalable while maintaining transparency through the inspectable constitution.

The core mechanism of the supervised phase is the critique-revision loop. Given a harmful query, the model:

1. Generates an initial response (which may be problematic).
2. Is prompted with a constitutional principle and asked to critique its own response, identifying specific violations.
3. Is prompted to revise the response to eliminate the identified violations. This iterative self-correction teaches the model to internalize the principles as an editing process rather than simply avoiding certain topics.

Unlike RLHF, where human preferences are opaque and difficult to audit, CAI's alignment objectives are explicitly codified in the constitution. This provides several advantages:

• Auditability: Engineers and ethicists can inspect, debate, and modify the exact principles governing model behavior.
• Debuggability: When the model exhibits unwanted behavior, developers can trace it to specific constitutional clauses and adjust them.
• Democratic Governance: The constitution can be developed through multi-stakeholder processes, making alignment a public, participatory process rather than a black-box corporate decision.

CAI encourages the model to engage in explicit ethical reasoning before producing a final output. When asked to critique a response, the model must articulate why a response violates a principle, not just that it does. This chain-of-thought process improves the quality of revisions and makes the model's moral reasoning observable. Researchers found that models trained with CAI are less evasive—they can engage with sensitive topics while maintaining harmlessness, rather than simply refusing to answer.

The predecessor and complement to CAI. RLHF trains a reward model on human preference comparisons, then uses reinforcement learning to align the language model. Key distinction: CAI replaces the human feedback loop for harmlessness with AI-generated feedback based on a constitution, reducing the scalability bottleneck and psychological burden of human annotation. RLHF remains critical for capturing nuanced human preferences that are difficult to codify in rules.

The broader category under which CAI falls. RLAIF uses an AI system—rather than humans—to generate the feedback signals for training. CAI is a specific instance of RLAIF where the AI feedback is governed by an explicit, human-written constitution. Other RLAIF approaches might use LLMs to generate preference data without a codified set of principles. This technique dramatically reduces the cost and time of alignment training.

A structured adversarial testing process where a dedicated team simulates malicious actors to probe an AI system for vulnerabilities, biases, and failure modes. Relationship to CAI: Red-teaming is used to discover gaps in the constitution itself—identifying harmful outputs the principles failed to prevent. The constitution is then iteratively revised to address these gaps, creating a continuous improvement loop between adversarial testing and principle refinement.

A core objective in AI alignment focused on preventing models from generating toxic, biased, dangerous, or otherwise harmful content. CAI's approach: Instead of relying on human labelers to identify harmful outputs, the model critiques its own responses against constitutional principles like 'choose the response that is least harmful.' This creates a scalable, transparent, and auditable harmlessness training pipeline that doesn't require exposing humans to disturbing content.

The initial stage of CAI training. The base language model is fine-tuned on high-quality demonstration data showing helpful and harmless responses. Role in CAI pipeline: SFT provides the foundational behavioral priors before the constitutional revision stage. The model first learns what good responses look like from examples, then the constitution-based RL stage refines these behaviors by having the model critique and revise its own outputs according to explicit principles.

An alternative alignment paradigm proposed by OpenAI where two AI agents debate a question, with a human judge determining the winner. Contrast with CAI: Debate relies on adversarial dynamics to surface truth, while CAI relies on a fixed set of principles for self-critique. Both aim to scale oversight beyond direct human evaluation, but CAI's constitution provides explicit, interpretable guardrails rather than relying on the emergent properties of adversarial argumentation.