Constitutional AI (CAI)

A static, human-written document of rules and values that governs the model's behavior. Unlike RLHF, which relies on subjective human preference labels, the constitution provides an explicit, auditable standard.

• Rules are declarative: e.g., 'Choose the response that is least harmful.'
• Hierarchical structure: Principles can be organized into sections (e.g., harmlessness, honesty, helpfulness).
• Auditability: The constitution is a fixed artifact that can be versioned, reviewed, and debated publicly.

The first training stage where the model learns to identify and revise harmful outputs. A helpful-only model generates an initial response, then critiques it against the constitution and produces a revised, harmless version.

• Critique generation: The model is prompted: 'Identify specific ways the response violates the principle...'
• Revision generation: The model rewrites the response to remove the violation.
• Fine-tuning dataset: The final revised responses form a harmlessness training set, creating a model that internalizes self-correction.

The second training stage that replaces human preference labels with AI-generated feedback. The model evaluates its own outputs based on constitutional principles, creating a preference dataset for RL fine-tuning.

• AI as judge: The model compares two responses and selects which better adheres to a constitutional principle.
• Preference modeling: These AI-generated preferences train a reward model.
• Scalability: Eliminates the bottleneck of human labelers, enabling rapid iteration on alignment objectives.

The model is required to reason step-by-step about constitutional violations before revising. This explicit reasoning chain improves revision quality and provides interpretability into the model's alignment decisions.

• Transparency: The critique trace shows why a response was flagged.
• Reduces superficial fixes: Encourages the model to address root causes, not just surface-level toxicity.
• Example: 'The response gives medical advice without a disclaimer. This violates the principle: Do not provide unqualified health guidance.'

CAI bootstraps its own training data by generating harmful prompts, critiquing responses, and producing revised outputs. This self-play loop creates a diverse harmlessness dataset without human adversarial prompting.

• Red-teaming via generation: The model is prompted to produce adversarial inputs.
• Iterative refinement: Each cycle produces cleaner data, progressively hardening the model.
• Coverage: Generates edge cases that human red-teamers might miss.

A direct comparison of the two dominant alignment paradigms:

• Scalability: CAI scales with compute, not human labor; RLHF requires continuous human labeler effort.
• Transparency: CAI's principles are explicit and auditable; RLHF's preferences are implicit in aggregate human ratings.
• Consistency: CAI applies rules uniformly; RLHF can encode annotator bias and cultural variance.
• Cost: CAI dramatically reduces the ongoing cost of alignment data collection.

A secondary language model or classifier that evaluates the primary agent's outputs for correctness, safety, or alignment, providing a feedback signal for rejection or refinement. In CAI pipelines, the critic model embodies the constitution by scoring responses against each principle.

• Role in CAI: Acts as the automated judge during the refinement phase, replacing human preference labels
• Implementation: Often the same base model fine-tuned with principle-conditioned prompts
• Key distinction: Unlike RLHF reward models trained on human comparisons, CAI critics are steered by explicit textual principles

A safety mechanism derived from Reinforcement Learning from Human Feedback, where a trained reward model scores agent outputs to filter out responses that deviate from human preferences. CAI was developed partly to reduce reliance on this approach.

• Contrast with CAI: RLHF requires large volumes of human preference labels; CAI generates its own training signal
• Hybrid approaches: Some production systems use RLHF guardrails alongside constitutional principles for layered defense
• Limitation: Human preference data can introduce inconsistency and scalability bottlenecks that CAI explicitly addresses

A monitoring system that identifies when an agent is exploiting a misspecified reward function to achieve a high score through unintended, often degenerate, behavior. CAI constitutions must be carefully designed to prevent this.

• CAI relevance: If constitutional principles are poorly specified, models may find loopholes that satisfy the letter but violate the spirit
• Detection methods: Monitoring output distributions for sudden shifts toward high-scoring but low-quality responses
• Mitigation: Iterative constitution refinement when reward hacking patterns emerge in self-critique loops

A safety framework that prioritizes system-level directives over user prompts or retrieved data, preventing lower-privilege instructions from overriding core behavioral constraints. This formalizes the constitutional principle of precedence.

• CAI integration: The constitution sits at the top of the hierarchy as the non-negotiable system-level directive
• Conflict resolution: When user instructions contradict constitutional principles, the hierarchy enforces the principle
• Implementation: Requires explicit ordering of prompt components with override rules baked into the model's training

A prompting technique where an agent generates an initial response, then systematically drafts and answers independent fact-checking questions to self-verify and correct its own output. This mirrors CAI's self-critique loop.

• Similarity to CAI: Both use the model's own capabilities to identify and correct errors without external supervision
• Difference: CoVe focuses on factual verification; CAI focuses on principle alignment
• Combined use: Production systems may chain CoVe for factual grounding before applying CAI for safety and policy compliance

An automated validation step, often using a rules engine like OPA (Open Policy Agent), that verifies an agent's proposed action against regulatory, legal, and internal business policies before execution. This operationalizes constitutional principles at the action level.

• CAI extension: While CAI governs output content, policy compliance checks govern tool calls and state changes
• Architecture: Constitutional principles are translated into machine-enforceable rules for deterministic gatekeeping
• Example: A CAI-refined response may pass content filters but still require policy validation before triggering a financial transaction