Constitutional AI

The model generates an initial response, then critiques it against a constitution of explicit rules (e.g., 'Choose the response that is least harmful'). This critique identifies specific violations, and the model revises its output accordingly. This process creates a dataset of self-corrected responses without human annotators needing to read disturbing content.

CAI operates in two distinct stages:

• Supervised Phase: The model critiques and revises its own harmful outputs using constitutional principles, generating a fine-tuning dataset of corrected responses.
• Reinforcement Learning Phase: The fine-tuned model generates pairs of responses, and a harmlessness classifier trained on the constitution's preferences selects the better one, further refining behavior via RL.

The constitution encodes human values into a set of interpretable, auditable rules that scale oversight without proportional human labor. Unlike RLHF, which requires humans to evaluate every harmful output, CAI's principles are applied programmatically. This allows for transparent governance—the exact rules shaping model behavior are public and modifiable.

A key safety innovation: human annotators in standard RLHF must read and rank deeply disturbing model outputs to train harmlessness. CAI eliminates this vicarious trauma by having the model itself perform the critique and revision. Humans only verify the final, revised outputs, dramatically improving annotator welfare.

Unlike the opaque preferences of a crowdworker population, a constitution is a public document open to scrutiny, debate, and iteration. Anthropic's original constitution includes principles from the UN Declaration of Human Rights, Apple's terms of service, and non-Western perspectives. This makes safety alignment an engineering discipline rather than a black-box sociological process.

CAI is typically used specifically for harmlessness training, while standard RLHF from human feedback is retained for helpfulness. This separation of concerns means the model learns to be useful from humans and safe from principles. The result is a model that maintains strong capabilities while exhibiting robust refusal behavior against jailbreak attempts.

The foundational alignment technique that Constitutional AI seeks to scale. Reinforcement Learning from Human Feedback trains a reward model based on human preferences for helpfulness and harmlessness. Constitutional AI replaces the human evaluator with an AI critic guided by a written constitution, drastically reducing the human annotation bottleneck while maintaining comparable safety outcomes.

The initial phase of Constitutional AI where the model generates self-critiques and revisions. The process:

• Model produces a harmful response to a red-teaming prompt
• Model critiques its own output according to constitutional principles
• Model revises the response to align with the constitution
• The resulting harmless pairs are used for SFT, teaching the model to internalize the principles directly.

The second phase of Constitutional AI that trains a harmlessness classifier without human labels. The AI generates pairs of responses to adversarial prompts, evaluates which best adheres to the constitution, and uses these AI-generated preferences to train a reward model. This synthetic preference data replaces the human feedback loop, enabling scalable oversight.

The adversarial practice of systematically probing a model to discover vulnerabilities. In Constitutional AI, automated red teaming generates diverse harmful prompts that the model must critique and revise. This creates a self-improving loop where the model learns to defend against increasingly sophisticated attacks, including many-shot jailbreaking and payload splitting.

A safety framework that establishes a privilege model for conflicting instructions. System-level constitutional principles receive highest priority, followed by user prompts, and lastly third-party data. This directly mitigates indirect prompt injection attacks where malicious instructions embedded in retrieved documents attempt to override the model's core behavioral constraints.

An alternative safety technique that manipulates internal model activations rather than training. By identifying and steering harmful concept vectors in the model's hidden states, safety can be enforced at inference time. This complements Constitutional AI by providing a runtime safety layer that doesn't require retraining the entire preference model.