Output Validation

Schema validation is the process of checking that a structured data object, such as JSON or XML, conforms to a predefined schema that specifies the required format, data types, and constraints. This is a foundational, deterministic check for API-based agents.

• Enforces Structure: Validates the presence of required fields, correct data types (string, integer, boolean), and nested object hierarchies.
• Prevents Integration Failures: Catches malformed outputs before they are passed to downstream tools or systems, preventing runtime errors.
• Common Tools: Implemented using libraries like Pydantic for Python, JSON Schema, or TypeScript interfaces.

Rule-based validation is a deterministic verification method where outputs are checked against a set of explicit, human-defined logical rules or conditions to ensure compliance with business logic.

• Explicit Logic: Uses if-then statements to enforce domain-specific rules (e.g., 'total cost must equal sum of line items', 'date must be in the future').
• Deterministic & Auditable: Provides clear pass/fail outcomes and an audit trail of which rule was triggered.
• Foundation for Guardrails: Often forms the core of guardrail systems that prevent unsafe or non-compliant actions.

This technique validates the meaning and factual correctness of an output, going beyond syntax to check if the content is grounded in source data and contextually accurate.

• Hallucination Detection: Identifies when an LLM produces confident but factually incorrect or unsupported information. Techniques include citation verification and embedding similarity checks against source documents.
• Semantic Validation: Ensures the output's intent aligns with the task (e.g., an extracted 'company name' field actually contains a company name, not a person's name).
• Context-Aware: Often requires cross-referencing the output with the agent's working memory or knowledge base.

A suite of checks designed to screen outputs for harmful, biased, or non-compliant content before they are exposed to users or external systems.

• Toxicity Detection: Uses ML classifiers to flag rude, disrespectful, or harmful language.
• PII Detection: Scans for Personally Identifiable Information (names, IDs, emails) to enforce privacy policies like GDPR.
• Bias Detection: Identifies skewed or unfair representations related to protected attributes.
• Prompt Injection Detection: Attempts to identify and block outputs that may contain hidden, malicious instructions from a compromised input.

Validation through direct code execution and comparison against known-good references, providing high-confidence verification for deterministic or repeatable tasks.

• Assertions: Code statements that check a condition (e.g., assert result['status'] in ['SUCCESS', 'FAILURE']). If false, the output is invalidated.
• Golden Tests: Compares the agent's output against a pre-approved, known-correct 'golden' reference output. Any deviation flags a potential regression or error.
• Syntax Validation: For code-generating agents, this involves checking that generated code compiles or passes linting rules.

Techniques that use probabilistic measures and statistical frameworks to assess the reliability of an output, particularly for non-deterministic model generations.

• Confidence Thresholds: A model's own probability score for its output is compared to a cutoff (e.g., 0.85). Outputs below the threshold are rejected or sent for human review.
• Conformal Prediction: A statistical framework that generates prediction sets with guaranteed error rates, providing rigorous, quantifiable uncertainty intervals.
• Ensemble Checking: Queries multiple models or prompts and validates output by measuring consensus or variance among the responses.

Validating that an LLM's output conforms to a strict JSON or XML schema is a foundational use case. This ensures downstream systems can parse the data without errors.

• Key Checks: Required fields, correct data types (string, integer, boolean), nested object structure, and enum value adherence.
• Example: An agent generating a customer support ticket must output a JSON with fields ticket_id (string), priority (enum: 'low', 'medium', 'high'), and description (string). Schema validation rejects outputs missing priority or with a numeric ticket_id.
• Tools: JSON Schema validators, Pydantic models, or Open Policy Agent (OPA) for policy-as-code validation.

Critical for Retrieval-Augmented Generation (RAG) systems, this validation ensures all factual claims in an output are supported by provided source documents.

• Key Checks: Citation verification (citations exist and are accurate), embedding similarity checks (output claims are semantically close to source text), and contradiction detection (output does not contradict source data).
• Example: A financial report generator cites a source document stating 'Q4 revenue was $5M.' Validation cross-references the citation; if the source says $4M, the output is flagged for hallucination.
• Method: Use a separate verification LLM call or embed both claim and source to compute cosine similarity, rejecting low-similarity outputs.

Preventing the generation of harmful, biased, or non-compliant content is a non-negotiable validation layer in production systems.

• Common Validations:
  • Toxicity Detection: Flagging outputs containing hate speech, harassment, or insults.
  • PII Detection: Scanning for and redacting personally identifiable information like credit card numbers or social security numbers.
  • Bias Detection: Identifying skewed representations based on gender, race, or other protected attributes.
  • Prompt Injection Detection: Identifying attempts to override system instructions via user input.
• Implementation: Often uses specialized classifiers (e.g., Perspective API for toxicity) or regex patterns for PII, acting as a circuit breaker to block unsafe outputs.

When an AI agent generates code (SQL, Python, shell commands), validation ensures it is syntactically correct and safe to execute.

• Key Checks:
  • Syntax Validation: Parsing the code with the language's interpreter/compiler to catch errors.
  • Static Analysis (SAST): Scanning for security vulnerabilities (e.g., SQL injection patterns, unsafe deserialization).
  • Sandboxed Execution: Running code in an isolated environment with limited permissions to verify it produces the expected result without side effects.
• Example: An agent generates a SQL query to fetch user data. Validation involves a dry-run syntax check, scanning for DROP TABLE or DELETE without a WHERE clause, and potentially executing it against a test database with a timeout.

Ensuring an output adheres to complex, domain-specific business rules that cannot be encoded in a simple schema.

• Key Checks: Validation against business rule engines or policy engines that evaluate logical conditions.
• Examples:
  • A loan approval agent must output a decision that complies with regulatory debt-to-income ratios.
  • A pricing agent's recommended discount must not exceed a manager's pre-approved authority limit.
  • A scheduling agent must not assign a worker more than 40 hours per week.
• Tools: Open Policy Agent (OPA) allows defining rules in Rego language. Validation passes the output context to OPA, which returns an allow/deny decision based on corporate policy.

In orchestrated systems, one agent's output becomes another's input. Validation ensures the data fulfills the expected 'contract' for a successful handoff.

• Key Checks: Schema validation for structure, plus semantic validation for meaning and completeness.
• Example: An Agent A researches a topic and must pass a summary to Agent B for writing. The validation contract requires a summary field (string, min 50 words) and a key_entities field (list). If Agent A's output lacks key_entities, the handoff fails, triggering a corrective action like re-prompting Agent A or rerouting the task.
• Pattern: This is a core component of fault-tolerant agent design, preventing cascading failures by validating inter-agent communication.

Schema validation is the process of checking that a structured data object, such as JSON, XML, or a Pydantic model, conforms to a predefined schema that specifies the required format, data types, and constraints. It is a foundational technique for ensuring deterministic output formatting from language models, crucial for reliable tool calling and API integration.

• Core Mechanism: Uses a formal schema definition (e.g., JSON Schema, OpenAPI) to validate the structure, data types (string, integer, array), required fields, and value ranges of an output.
• Use Case: Essential for structured generation, where an LLM must output valid, parsable data for downstream systems. A failed validation triggers a retry or correction loop.
• Tools: Libraries like Pydantic (Python), Zod (TypeScript), and JSON Schema validators are standard in agentic pipelines to enforce contract-first development.

Semantic validation is the process of checking that the meaning or intent of an output is correct and consistent with its context, going beyond simple syntactic or format checks. It answers the question: "Is this output logically and factually correct given the task?"

• Techniques: Involves natural language inference (NLI), entailment checks, knowledge graph querying, and embedding similarity checks to compare the generated content against trusted sources or the user's intent.
• Contrast with Syntax: While syntax validation ensures a JSON key exists, semantic validation ensures the value of that key makes sense (e.g., a temperature value is within a plausible range for a weather report).
• Challenge: Requires a reference truth or a set of business logic rules to evaluate against, making it more complex to automate than structural validation.

Hallucination detection is the process of identifying when a generative AI model, particularly a large language model, produces confident but factually incorrect or nonsensical information not grounded in its source data. It is a critical validation step for Retrieval-Augmented Generation (RAG) systems and any application requiring factual accuracy.

• Methods:
  • Citation Verification: Checking if generated statements are supported by provided source citations.
  • Embedding-Based Fact-Checking: Comparing the claim's embedding to the embeddings of source chunks to measure semantic consistency.
  • Contradiction Detection: Using a model to identify if two statements (the claim and the source) contradict each other.
• Output: Typically produces a confidence score or a binary flag, which can trigger a regeneration request or a human-in-the-loop review.

Rule-based validation is a deterministic verification method where outputs are checked against a set of explicit, human-defined logical rules or conditions to ensure compliance. It is the most transparent and auditable form of validation, ideal for enforcing business rules, safety policies, and regulatory compliance.

• Characteristics: Rules are if-then statements or pattern-matching expressions (e.g., regex). They provide clear, explainable pass/fail criteria.
• Examples:
  • "If the transaction amount > $10,000, then the risk_level field must be 'HIGH'."
  • "The generated summary must contain the keywords 'Q1' and 'revenue'."
  • "The output must not match the regex pattern for a social security number."
• Integration: Often implemented as a series of assertions in a validation pipeline, acting as the first line of defense before more complex, model-based checks.

A validation pipeline is an automated, multi-stage workflow that applies a series of checks and tests to system outputs to ensure they meet quality, safety, and functional requirements before being accepted. It orchestrates various validation techniques into a cohesive quality gate.

• Typical Stages:
  1. Syntax & Schema Check: Validates structure and format.
  2. Rule-Based Check: Applies business logic and safety rules.
  3. Semantic/Model-Based Check: Uses classifiers or LLMs for factual, toxicity, or bias detection.
  4. Integration Test: Verifies the output works correctly with downstream systems.
• Design Pattern: Often follows a filter chain or middleware pattern, where an output must pass all stages. A failure at any stage can trigger a retry, correction, or escalation.
• Tools: Built using workflow orchestrators (Airflow, Prefect), ML pipelines (Kubeflow), or custom frameworks incorporating libraries for each validation type.