Output Format Drift Detection Prompt Template

This prompt is designed for platform engineers and AI SREs who need to detect when production model outputs begin drifting away from their expected schema. Format drift is a leading indicator of prompt degradation, model version behavioral changes, or context-window corruption. Use this prompt when you have a defined expected output schema and a batch of recent production outputs that need compliance scoring. The prompt produces a structured drift report with an overall drift score, field-level violation counts, and a regression alert when drift exceeds acceptable thresholds. This is not a prompt for initial schema design or output repair. It assumes outputs already exist and need to be audited against a reference contract.

The ideal user is an engineer responsible for production AI quality who already has access to a sample of recent model responses and a formal schema definition (JSON Schema, Pydantic model, or a strict type specification). The prompt requires two concrete inputs: a reference [EXPECTED_SCHEMA] that defines the correct output contract, and a [PRODUCTION_OUTPUT_SAMPLE] containing the actual model responses to audit. Without both, the prompt cannot produce a meaningful drift report. The prompt is most effective when run as a scheduled batch job—daily or per-deployment—comparing outputs against a known-good schema version. It is not a substitute for real-time validation in your application layer; rather, it is a monitoring tool that catches degradation your inline validators might miss due to lenient parsing or partial schema enforcement.

Do not use this prompt when you need to repair malformed outputs, design a new schema, or debug a single bad response. For output repair, use a dedicated repair prompt that takes the malformed output and the expected schema as inputs. For schema design, use a structured output design prompt that iterates on field definitions and constraints. For single-response debugging, a trace review prompt that inspects the full request context—system prompt, tools, retrieval results—will be more diagnostic than a batch drift report. This prompt is also not suitable for evaluating semantic quality, factual accuracy, or tone; it only checks structural compliance. Pair it with an LLM Judge or eval rubric prompt when you need to assess both format and content quality in the same monitoring pipeline.

This prompt template is designed to be integrated into your continuous monitoring pipeline. It compares a production LLM output against a reference schema and a previous stable version to detect format drift. The prompt produces a structured drift report that includes a drift score, field-level violation details, and a regression alert flag. Use this when you need automated, trace-level detection of schema compliance degradation before it impacts downstream consumers.

textCopy
You are an output format drift detector for a production AI system. Your task is to compare the provided [CURRENT_OUTPUT] against the [EXPECTED_SCHEMA] and, if available, the [BASELINE_OUTPUT] from a previous stable version. You must produce a structured drift report.

## Inputs
- Expected Schema: [EXPECTED_SCHEMA]
- Current Output: [CURRENT_OUTPUT]
- Baseline Output (optional): [BASELINE_OUTPUT]
- Drift Sensitivity: [DRIFT_SENSITIVITY] (e.g., 'strict', 'moderate', 'lenient')

## Instructions
1. Parse the [CURRENT_OUTPUT] and validate its structure against the [EXPECTED_SCHEMA].
2. Identify any missing required fields, extra fields, type mismatches, or enum violations.
3. If [BASELINE_OUTPUT] is provided, compare the structure of [CURRENT_OUTPUT] to [BASELINE_OUTPUT] and flag any structural regressions (e.g., a field that was present in the baseline is now missing or has changed type).
4. Calculate a drift score from 0.0 (no drift) to 1.0 (complete structural failure) based on the number and severity of violations.
5. Set a boolean regression alert to true if any structural regression is detected against the baseline.

## Output Format
Return a single valid JSON object with the following schema:
{
  "drift_score": <float 0.0-1.0>,
  "regression_alert": <boolean>,
  "schema_violations": [
    {
      "field_path": "<string>",
      "violation_type": "<missing_required | extra_field | type_mismatch | enum_violation>",
      "expected": "<string>",
      "actual": "<string>",
      "severity": "<critical | warning>"
    }
  ],
  "baseline_regressions": [
    {
      "field_path": "<string>",
      "baseline_type": "<string>",
      "current_type": "<string>"
    }
  ],
  "summary": "<A concise, human-readable summary of the drift analysis.>"
}

## Constraints
- Do not modify or repair the [CURRENT_OUTPUT]; only analyze it.
- If [BASELINE_OUTPUT] is not provided, omit the 'baseline_regressions' array and set 'regression_alert' to false.
- Treat any deviation from the [EXPECTED_SCHEMA] as a violation, with severity based on [DRIFT_SENSITIVITY].

To adapt this template for your environment, replace the square-bracket placeholders with data from your monitoring pipeline. The [EXPECTED_SCHEMA] should be a JSON Schema or a detailed natural language description of the required output format. The [CURRENT_OUTPUT] is the raw string from your LLM's latest response. The optional [BASELINE_OUTPUT] should be a previously validated, correct output from an earlier prompt version or model. Wire this prompt into a post-processing step that runs after every production response, and log the resulting drift report as a structured trace span. For high-stakes applications, pair this automated check with a human review step when the drift score exceeds a threshold like 0.3 or when a regression alert is triggered.

The Output Format Drift Detection prompt is designed to run as a scheduled evaluation job, not as a synchronous part of the user-facing request path. It compares a sample of recent production outputs against a reference schema and a baseline of previously accepted outputs. The harness must pull these inputs from your observability store, invoke the prompt, validate the structured drift report it returns, and route any detected regressions to the appropriate alerting channel. Because this prompt performs a statistical comparison, you should run it on a rolling window (e.g., the last 1,000 responses) rather than on individual outputs, and you should configure the window size and sampling strategy based on your traffic volume and SLO sensitivity.

The implementation loop follows a strict sequence: fetch a batch of recent outputs and the reference schema from your trace store or logging backend, assemble the prompt with the [EXPECTED_SCHEMA], [BASELINE_OUTPUTS], and [CURRENT_OUTPUTS] placeholders populated, invoke the model with a low temperature (0.0–0.1) to maximize deterministic scoring, validate the returned JSON against a drift report schema that includes drift_score, field_level_violations, and regression_alert fields, retry once if validation fails with a more explicit schema reminder, and route the validated report to your alerting system if regression_alert.triggered is true. Each step must be logged with the prompt version, model version, window parameters, and the raw drift report for auditability. If the prompt fails validation twice, escalate to a human reviewer with the raw outputs attached rather than silently dropping the evaluation.

The most common production failure mode is schema validation of the prompt's own output. The model may return a drift report with extra commentary, missing fields, or malformed JSON when the input batch is large or contains unusual edge cases. Mitigate this by using a strict JSON mode or structured output API if your model provider supports it, and by implementing a post-processing validator that checks for required keys (drift_score, field_level_violations, regression_alert, comparison_window, evaluation_timestamp) before the report enters your alerting pipeline. A second failure mode is false-positive drift alerts caused by intentional schema evolution—if your product team deliberately added a new optional field, the prompt will flag it as a format addition. Wire the harness to accept a [SCHEMA_CHANGE_WINDOW] parameter that suppresses alerts for known, approved schema changes during a migration period. Finally, never use this prompt's output to automatically block or reject user-facing responses; its role is observability and alerting, not enforcement.