Constitutional AI

The initial phase of CAI involves generating self-critiques and revisions based on a constitution—a static list of natural language principles. The model is prompted to generate a harmful response to a red-team prompt, then asked to critique its own output according to a randomly sampled principle, and finally asked to rewrite the original response to remove the identified harm. This process generates a fine-tuning dataset of revised, harmless responses without any human labels on harmlessness. The resulting model learns to internalize the critique process, reducing the need for human evaluators to constantly review disturbing content.

The second phase replaces human preference data with RLAIF (Reinforcement Learning from AI Feedback). The fine-tuned model from Phase 1 generates a pair of responses to a harmful prompt. The model is then asked to evaluate which response is better according to a constitutional principle, producing an AI-generated preference label. These preferences train a reward model, which is then used to further fine-tune the policy via reinforcement learning. This creates a fully automated alignment loop where the constitution acts as the sole source of normative guidance, dramatically improving scalability.

The constitution is a transparent, auditable document containing high-level principles sourced from multiple origins:

• UN Declaration of Human Rights: Principles on dignity, privacy, and non-discrimination.
• AI Safety Guidelines: Rules against generating instructions for weapons, hacking, or illegal acts.
• Platform Content Policies: Standards derived from trust and safety best practices.
• Anthropic's Own Research: Principles encouraging helpfulness, honesty, and epistemic humility. Because the principles are explicit text, they can be inspected, debated, and modified by stakeholders, making the alignment process far more transparent than opaque human preference data.

CAI addresses two critical bottlenecks in standard RLHF (Reinforcement Learning from Human Feedback). First, it eliminates the human welfare cost of exposing annotators to toxic content for harmlessness training. Second, it makes alignment transparent: the constitution is a public artifact that can be audited, version-controlled, and democratically debated. This contrasts sharply with RLHF, where human preferences are a black box of aggregated subjective judgments. CAI also enables rapid iteration—updating a principle in the constitution immediately changes the model's behavior without a new human labeling campaign.

During both training and inference, CAI models are encouraged to engage in chain-of-thought reasoning about the constitution before finalizing an output. The model explicitly considers which principles are relevant to the query, identifies potential violations in a draft response, and articulates a revision strategy. This internal monologue is not shown to the end user but serves as a structured reasoning scaffold. This process improves the quality of revisions and provides a form of interpretability, as the model's ethical reasoning trace can be inspected by developers for debugging and auditing purposes.

CAI models exhibit a distinct robustness profile against jailbreaking and prompt injection attacks. Because the model has internalized a set of principles through iterative self-critique rather than surface-level pattern matching, it is often more resistant to adversarial prompts that attempt to bypass safety training. However, CAI is not immune—determined red teaming can still uncover edge cases where constitutional principles conflict or are ambiguously applied. This has led to ongoing research into principle refinement and the use of adversarial training specifically targeting constitutional reasoning chains.

A structured adversarial exercise where a dedicated team simulates real-world attacks to proactively identify vulnerabilities, safety failures, and security gaps. In the Constitutional AI pipeline, red teaming is used to discover edge cases where the model's self-critique fails, generating adversarial prompts that are then incorporated into the refinement dataset to strengthen the constitution's coverage.

Programmatic constraints and validation layers integrated into an AI application's runtime to enforce safety policies. While Constitutional AI shapes the model's internal behavior during training, guardrails act as a runtime safety net:

• Input guardrails filter malicious prompts before they reach the model
• Output guardrails block or rewrite non-compliant generations
• Together they provide defense-in-depth when constitutional principles fail

An extension of the Constitutional AI paradigm where the human-written constitution is replaced by AI-generated feedback. The model critiques and revises its own outputs based on principles, and these AI-generated preference labels are used to train the reward model. This fully removes the human bottleneck from the alignment pipeline, enabling scalable oversight entirely through automated self-improvement.

A defensive technique that improves model robustness by augmenting the training dataset with adversarial examples. In the Constitutional AI context, the model's own red-teamed outputs—where it successfully jailbreaks itself—become the adversarial training data. The model learns to recognize and reject its own failure modes, hardening the constitution against circumvention attempts.

The fundamental alignment research problem that Constitutional AI addresses: how to supervise AI systems that surpass human capabilities. When models become too capable for humans to reliably evaluate, written constitutions provide a mechanism for the model to supervise itself. This shifts oversight from evaluating outputs to specifying principles, which scales more efficiently than human feedback loops.