Schema Conformance Eval Prompt Template

This playbook is built for platform teams and AI quality engineers who need to validate structured outputs before they reach downstream systems. The primary job-to-be-done is field-level error reporting: you have a candidate output and a reference schema, and you need a detailed conformance report that identifies missing required keys, type mismatches, and nested-structure violations. The ideal user is someone integrating this evaluation into a CI/CD pipeline, a regression test suite, or a production guardrail where structured output integrity is a hard requirement. You should already have both the candidate output and the schema available; this prompt does not generate conformant outputs from scratch.

Do not use this prompt when you need to repair malformed outputs, generate new conformant data, or evaluate semantic correctness rather than structural conformance. It is not a substitute for output repair prompts, synthetic data generation, or factuality evaluation. The prompt assumes the schema is well-formed and the candidate output is parseable; if the candidate output is not valid JSON, the evaluation will fail at the parsing stage before the LLM can produce a report. For high-risk domains such as finance, healthcare, or legal applications, always pair this evaluation with human review of the conformance report before allowing outputs to reach production systems.

Before using this prompt, ensure you have a versioned schema and a representative set of candidate outputs that exercise edge cases: deeply nested objects, arrays with mixed types, optional fields that are sometimes present and sometimes absent, and boundary values like null, empty strings, and zero. The prompt is designed to be used as a gate in automated pipelines, so plan to integrate its output with your existing validation harness, logging infrastructure, and alerting systems. Start by running it against known-good and known-bad outputs to calibrate your pass/fail thresholds before deploying it as a production guardrail.

This template is the core instruction set for an LLM-based schema validator. It is designed to be dropped into your evaluation harness, where you will replace the square-bracket placeholders with the specific schema, the output to be tested, and any custom validation rules. The prompt instructs the model to act as a strict, deterministic validator, not a conversational assistant. Its primary job is to compare the [OUTPUT_TO_VALIDATE] against the [TARGET_SCHEMA] and produce a structured [CONFORMANCE_REPORT].

textCopy
You are a strict schema validation engine. Your only task is to validate the provided [OUTPUT_TO_VALIDATE] against the [TARGET_SCHEMA]. Do not correct the output. Do not provide conversational advice.

Follow these steps precisely:
1. Parse the [TARGET_SCHEMA] to understand all required fields, their types, allowed enum values, and nested object structures.
2. Parse the [OUTPUT_TO_VALIDATE].
3. Perform a field-by-field comparison, checking for:
   - Missing required keys.
   - Type mismatches (e.g., string vs. integer).
   - Values not in an allowed enum list.
   - Incorrectly structured nested objects or arrays.
   - Additional keys not defined in the schema (if [ADDITIONAL_PROPERTIES_POLICY] is 'forbidden').
4. For each error found, record the exact path to the field, the expected condition, and the actual value found.

[ADDITIONAL_VALIDATION_RULES]

Generate the output strictly as a JSON object conforming to the [CONFORMANCE_REPORT_SCHEMA] below. Do not include any text outside the JSON object.

[CONFORMANCE_REPORT_SCHEMA]
{
  "valid": boolean,
  "errors": [
    {
      "path": "string (e.g., 'user.address.city')",
      "rule_violated": "string (e.g., 'required', 'type', 'enum')",
      "expected": "string",
      "actual": "string"
    }
  ]
}

[TARGET_SCHEMA]
[OUTPUT_TO_VALIDATE]

To adapt this template, start by replacing the [TARGET_SCHEMA] and [OUTPUT_TO_VALIDATE] placeholders with your actual JSON schema and the model's generated output. The [ADDITIONAL_VALIDATION_RULES] placeholder is where you inject custom business logic, such as "The 'id' field must be a UUID v4" or "The 'start_date' must be before the 'end_date'". The [ADDITIONAL_PROPERTIES_POLICY] variable should be set to either 'forbidden' or 'allowed' to control strictness. For high-stakes applications, always pair this prompt with a deterministic post-processing step: parse the generated [CONFORMANCE_REPORT] with a standard JSON library and re-run critical type and structure checks in application code to guard against LLM non-determinism in the validation report itself.

The Schema Conformance Eval prompt is designed to be a single step in a larger validation harness, not a standalone script. You should call it after your primary generation step produces a candidate output. The prompt expects a [TARGET_OUTPUT] (the string to validate) and a [SCHEMA_DEFINITION] (a JSON Schema, TypeScript interface, or structured data contract). The model returns a structured conformance report, which your application must parse and act on. Do not treat the model's verdict as the final gate; instead, use it as a high-signal diagnostic that feeds into deterministic post-checks.

Wire this prompt into a retry loop with a strict maximum of 2-3 attempts. On each failure, pass the previous conformance report's errors array back to a repair prompt (see the Output Repair and Validation Prompts pillar) before re-validating. After the final attempt, if errors remain, log the full conformance report, the raw output, and the schema version to your observability platform. For high-risk domains like finance or healthcare, route any output with severity: "critical" errors to a human review queue and block downstream consumption. Use a model with strong JSON mode and structured output support (e.g., GPT-4o, Claude 3.5 Sonnet) for the eval step itself, as weaker models may hallucinate error locations.

Before deploying, build a small golden dataset of 20-50 examples: include perfectly conformant outputs, outputs with missing required fields, type mismatches, and deeply nested errors. Run the eval prompt against this dataset and assert that the valid boolean matches your ground truth in at least 95% of cases. Pay special attention to false negatives (marking valid outputs as invalid), which will stall your pipeline. Also test with schemas containing $ref, oneOf, and additionalProperties: false to ensure the eval prompt handles complex JSON Schema features. If you observe flaky results, consider running the eval twice and requiring agreement before failing an output.