Format Drift Eval Failure Diagnosis Prompt

This playbook is for platform engineers and AI reliability engineers who operate production pipelines with schema-enforced outputs. When automated structured output evaluations begin failing, the root cause often falls into one of two categories: the model produced the wrong content, or the model produced content in the wrong shape. This prompt addresses the second category—format drift—by comparing an actual model output against an expected JSON schema and producing a field-level compliance report. Use it as a first diagnostic step before diving into content-quality investigations, hallucination checks, or retrieval pipeline debugging. If the format is intact, you can confidently shift focus to semantic or factual issues.

The prompt expects three inputs: the actual model output that failed evaluation, the expected JSON schema (as a valid JSON Schema object), and any field-level descriptions or constraints that clarify intent. It returns a structured compliance report identifying missing required keys, type violations (e.g., string where an integer was expected), extra fields not declared in the schema, and structural changes such as nested object flattening or array wrapping. The report includes a format_valid boolean, a failure_category classification (e.g., missing_key, type_mismatch, extra_field, structural_change), and a per-field breakdown with the expected type, actual type, and a human-readable diagnosis. This output is designed to be machine-readable so you can wire it into automated triage pipelines that route format failures to prompt repair and content failures to deeper trace analysis.

Do not use this prompt when you already know the output format is correct but the content is wrong—that requires a different diagnostic path, such as the Golden Set Deviation Diagnosis Prompt or the Hallucination Spot-Check Against Eval Rubric Prompt. Also avoid using it for non-JSON outputs unless you adapt the schema comparison logic to the target format (XML, YAML, etc.). For high-risk domains where format errors could cause downstream data corruption or compliance violations, always pair this prompt's output with human review before applying automated fixes. The compliance report is a diagnostic tool, not a repair mechanism—use the Output Repair and Validation Prompts pillar for actual format correction workflows.

Use this prompt to investigate why a structured output evaluation is failing. It is designed for platform engineers who need to compare an actual model output against an expected schema and produce a detailed compliance report. The prompt isolates field-level drift, type violations, missing or extra keys, and structural changes that cause format eval failures. Before pasting, ensure you have the actual output, the expected schema, and any relevant trace data showing the prompt and model version that generated the output.

textCopy
You are a schema compliance auditor for production AI systems. Your task is to compare an actual structured output against an expected schema and produce a detailed compliance report identifying field-level drift, type violations, and structural changes.

## INPUT

[ACTUAL_OUTPUT]

## EXPECTED SCHEMA

[EXPECTED_SCHEMA]

## TRACE CONTEXT

[TRACE_CONTEXT]

## INSTRUCTIONS

1. Parse the actual output and the expected schema.
2. Compare every field, type, and structural element.
3. Classify each finding into one of these categories:
   - TYPE_VIOLATION: field exists but has wrong type
   - MISSING_REQUIRED: required field absent from output
   - EXTRA_FIELD: field present in output but not in schema
   - STRUCTURAL_CHANGE: nesting, array, or object shape differs
   - ENUM_VIOLATION: value not in allowed enum set
   - NULL_VIOLATION: null value where non-null required
   - FORMAT_DRIFT: valid but unexpected format change (e.g., date format, casing)
4. For each finding, include:
   - The field path (dot notation)
   - Expected vs. actual value or type
   - Severity: CRITICAL, HIGH, MEDIUM, LOW
   - Likely root cause hypothesis based on trace context
5. Produce a summary with:
   - Total findings count by severity
   - Overall compliance score (0-100)
   - Top 3 recommended fixes
   - Whether this is likely a prompt regression, model behavior change, or parsing error

## OUTPUT FORMAT

Return a JSON object with this exact structure:
{
  "compliance_score": number,
  "findings": [
    {
      "field_path": "string",
      "category": "TYPE_VIOLATION | MISSING_REQUIRED | EXTRA_FIELD | STRUCTURAL_CHANGE | ENUM_VIOLATION | NULL_VIOLATION | FORMAT_DRIFT",
      "expected": "string",
      "actual": "string",
      "severity": "CRITICAL | HIGH | MEDIUM | LOW",
      "root_cause_hypothesis": "string"
    }
  ],
  "summary": {
    "total_findings": number,
    "by_severity": {
      "critical": number,
      "high": number,
      "medium": number,
      "low": number
    },
    "top_recommendations": ["string"],
    "likely_failure_category": "PROMPT_REGRESSION | MODEL_BEHAVIOR_CHANGE | PARSING_ERROR | UNKNOWN"
  }
}

## CONSTRAINTS

- Do not hallucinate fields not present in the actual output.
- If the actual output is not valid JSON, report a CRITICAL finding and stop.
- Base root cause hypotheses only on evidence in the trace context.
- If trace context is insufficient, set root_cause_hypothesis to "INSUFFICIENT_TRACE_DATA".

To adapt this prompt, replace [ACTUAL_OUTPUT] with the raw model response that failed evaluation, [EXPECTED_SCHEMA] with your target schema definition (JSON Schema, TypeScript interface, or plain description), and [TRACE_CONTEXT] with relevant trace data such as the system prompt, model version, and any tool calls or retrieval steps. If you lack trace data, the prompt will still produce findings but root cause hypotheses will be marked as insufficient. For high-risk domains like healthcare or finance, always route the compliance report through human review before taking action on the findings.

This prompt is designed to be the analytical core of an automated schema compliance investigation, not a one-off chat interaction. To use it effectively, you must embed it within a harness that feeds it structured trace data, validates its output, and routes the resulting report to the right team. The prompt expects a specific input shape: the [EXPECTED_SCHEMA] (a JSON Schema or TypeScript interface), the [ACTUAL_OUTPUT] (the raw string the model produced), and any [PARSER_ERRORS] from your initial validation layer. The harness's job is to gather these inputs from your production logging or eval pipeline and present them cleanly to the model.

The implementation should follow a strict validate → diagnose → route pattern. First, your application's existing output parser should attempt to validate the raw model output against the expected schema. If validation fails, capture the exact error message and the raw output string. The harness then calls this prompt with those three inputs. The model's response, a structured JSON report, must itself be validated against a strict schema (e.g., requiring drift_category, field_analysis, and root_cause_hypothesis fields) before being trusted. A retry loop with a maximum of 2 attempts is recommended if the diagnosis output fails its own schema validation, using a temperature of 0 to minimize variance. Log the raw diagnosis report and the final validated report as separate events in your trace for auditability.

For high-stakes systems where format drift could cause downstream data corruption or customer-facing errors, do not automatically apply the prompt's suggested fix. Instead, route the validated diagnosis report to a human review queue or a prompt-version-control system. The report's severity field can be used as a routing key: CRITICAL or HIGH severity findings should trigger an immediate alert and block further automated deployments until a human confirms the root cause and approves a remediation plan. This harness turns a diagnostic prompt into a safe, auditable control point in your MLOps pipeline, preventing a single malformed output from cascading into a full-scale regression.