Critic Model

The simplest critic implementation uses a classifier head on top of a pre-trained language model to output a probability score indicating whether the primary agent's response is acceptable or should be rejected. These models are fine-tuned on paired datasets of valid and invalid outputs, often using cross-entropy loss against human-labeled judgments. The output is a single logit transformed via sigmoid activation, producing a confidence score between 0 and 1 that feeds directly into a confidence threshold gate.

• Common base architectures: DeBERTa-v3, RoBERTa-large
• Typical training data: 10k-100k labeled (prompt, response, judgment) triplets
• Inference latency: < 50ms on A10G GPUs for real-time gating

Rather than training a dedicated classifier, this approach uses a general-purpose LLM prompted to evaluate another model's output against a rubric. The critic receives the original prompt, the primary agent's response, and a set of evaluation criteria, then generates a structured assessment—often using constrained decoding to produce valid JSON with fields for score, reasoning, and pass/fail determination. This pattern enables rapid iteration on evaluation criteria without retraining.

• Common evaluator models: GPT-4, Claude 3.5 Sonnet, Llama 3 70B
• Rubric dimensions: factual accuracy, safety compliance, instruction adherence
• Key risk: critic hallucination requires Chain-of-Verification on assessments

Derived from Constitutional AI (CAI), this architecture chains multiple critic passes where the model self-evaluates its output against a predefined set of principles—the 'constitution'—and iteratively revises until all constraints are satisfied. Each principle acts as a binary classifier: the critic checks if the output violates a specific rule, and if so, generates a critique and revision. This creates a self-improving feedback loop without external human labels.

• Principles cover: harmlessness, honesty, helpfulness, format compliance
• Typical loop depth: 2-4 revision iterations before timeout
• Integration point: sits between primary generation and output sanitization

A critic trained via Reinforcement Learning from Human Feedback (RLHF) that outputs a scalar reward representing human preference alignment. Unlike binary classifiers, reward models produce continuous scores that capture nuanced quality gradients—enabling ranking of multiple candidate responses rather than simple accept/reject decisions. These models are typically trained on paired comparison data where human annotators indicate which of two responses is preferred.

• Architecture: transformer encoder with linear regression head
• Training objective: Bradley-Terry preference model loss
• Use case: selecting best-of-N samples or weighting ensemble outputs

Critics that evaluate outputs across modalities—text, images, audio, or video—using modality-specific encoders fused into a unified safety classifier. A vision-language model critic might check generated images for policy violations while a text critic simultaneously evaluates accompanying captions. These systems often employ late fusion architectures where modality-specific embeddings are concatenated before final classification.

• Image safety dimensions: NSFW, violence, CSAM, graphic content
• Common fusion approach: CLIP embeddings + text embeddings → MLP classifier
• Critical for embodied agents with Vision-Language-Action outputs

Advanced critics that don't just output a judgment but also quantify their own epistemic uncertainty about that judgment. Techniques include Monte Carlo dropout during inference, deep ensembles of critic models, and conformal prediction sets that provide statistical guarantees on error rates. When uncertainty is high, the system can escalate to a human-in-the-loop reviewer rather than making an unreliable automated decision.

• Ensemble size: 3-5 independently trained critic models
• Conformal prediction: provides prediction sets with 90-99% coverage guarantees
• Integration: feeds into uncertainty quantification pipelines for gating decisions

Anthropic's Constitutional AI (CAI) framework uses a critic model to evaluate and revise outputs against a predefined set of principles. The primary model generates an initial response, then the critic identifies violations and proposes revisions.

• Supervised phase: Critic critiques responses based on the constitution
• RL phase: Critic-derived preference data trains a reward model
• Result: Alignment without heavy reliance on human RLHF labels

This approach enables scalable oversight where the critic enforces safety constraints autonomously.

Using a stronger or equally capable language model as a critic to score outputs on dimensions like helpfulness, harmlessness, and factual accuracy. This paradigm powers automated evaluation pipelines.

• Pairwise comparison: Critic selects the better of two candidate responses
• Reference-grounded: Critic checks outputs against provided ground truth
• Rubric-based: Critic scores against detailed grading criteria

Platforms like MT-Bench and AlpacaEval rely on LLM judges to approximate human preferences at scale, though position bias and verbosity bias require careful mitigation.

In Reinforcement Learning from Human Feedback (RLHF), a trained reward model acts as a critic that assigns scalar scores to agent outputs. This score guides policy optimization.

• Training: Reward model learns from human preference comparisons
• Inference: Scores reflect alignment with human values
• Application: Filters low-scoring outputs before user exposure

The reward model critic is central to aligning models like GPT-4 and Claude, serving as a differentiable proxy for human judgment during fine-tuning.

A self-critique prompting technique where the agent generates an initial response, then systematically drafts and answers independent fact-checking questions to verify its own output.

• Plan verification questions: Critic decomposes claims into checkable facts
• Execute verification: Critic answers each question independently
• Final response: Original output is corrected based on verification results

CoVe reduces hallucination rates by forcing the model to act as its own fact-checking critic before delivering the final answer to users.

Specialized classifier-based critic models that screen generated text for policy violations before delivery. These operate as lightweight, fast inference filters.

• Perspective API: Detects toxicity, threats, and profanity
• Llama Guard: Classifies safety risks across multiple harm categories
• Azure AI Content Safety: Multimodal filtering for text and images

These critics run in parallel with generation, providing real-time gating with minimal latency overhead. Outputs flagged above threshold are blocked or routed for human review.

Critic models that evaluate agent-generated code for correctness, security vulnerabilities, and policy compliance before execution in sandboxed environments.

• Static analysis: Critic checks for injection patterns and unsafe imports
• Test case generation: Critic produces unit tests to validate logic
• Output prediction: Critic estimates expected results for comparison

Systems like SWE-bench evaluation harnesses use critic models to verify that agent-generated patches actually resolve reported issues without introducing regressions.

A programmatic policy or mechanism that constrains an AI agent's behavior to prevent harmful, off-policy, or unsafe actions and outputs. Guardrails can be implemented as rule-based filters, classifier models, or LLM-based critics that evaluate outputs against predefined safety taxonomies. Unlike a general critic model, guardrails often enforce hard constraints—blocking outputs outright rather than providing nuanced feedback for refinement.

An alignment method developed by Anthropic that trains models to self-critique and revise their outputs based on a predefined set of principles, or a 'constitution', without heavy reliance on human RLHF labels. The model acts as its own critic, evaluating responses against rules like 'choose the response that is least harmful.' This creates an intrinsic feedback loop where the critic and generator are the same model, contrasting with architectures that deploy a separate critic model for external evaluation.

A quantitative metric that estimates the degree of factual inconsistency in a generated response. Common approaches include:

• Natural Language Inference (NLI) models that detect contradictions between the output and source documents
• Semantic similarity measurements against ground truth
• SelfCheckGPT techniques that sample multiple generations and measure consistency Critic models often use hallucination scores as a primary signal for rejection or triggering regeneration.

A control point in an agentic workflow that requires explicit validation or approval before a high-stakes tool call or state-changing operation is executed. The critic model serves as an automated gatekeeper at these checkpoints, evaluating the proposed action against safety policies, business rules, and alignment criteria. If the critic's confidence falls below a threshold, the action is escalated to a Human-in-the-Loop (HITL) reviewer.

A safety mechanism derived from Reinforcement Learning from Human Feedback, where a trained reward model scores agent outputs to filter out responses that deviate from human preferences. The reward model functions as a specialized critic, trained on pairwise comparison data where humans ranked outputs by quality, helpfulness, and harmlessness. This approach captures nuanced human values that are difficult to encode in explicit rules.

A set of statistical methods used to measure the model's confidence in its predictions, enabling the system to abstain from acting on low-certainty outputs. Techniques include:

• Conformal prediction for generating prediction sets with guaranteed coverage
• Monte Carlo dropout for Bayesian uncertainty estimates
• Semantic entropy for measuring meaning-level uncertainty in LLM outputs Critic models leverage uncertainty signals to decide when to reject or flag outputs for review.