Cross-Model Output Consistency Against Reference Prompt Template

This prompt is designed for engineering teams running multi-model systems that must produce consistent, predictable outputs. The core job-to-be-done is detecting divergence: when you have a shared reference answer and need to know whether Claude, GPT-4o, Gemini, and your fine-tuned Llama model all produce outputs that align with that reference in the same way. This is not a general quality evaluation prompt. It is specifically for regression testing, model migration validation, and multi-model deployment monitoring where consistency against a known-correct answer is the primary signal you care about.

Use this prompt when you are: (1) validating that a new model version doesn't break output patterns that downstream systems depend on, (2) running a multi-model deployment where requests can be routed to different models and outputs must be interchangeable, (3) maintaining a golden dataset of reference answers and need to detect which models drift from expected behavior, or (4) building a CI/CD pipeline where prompt changes must be tested across all target models simultaneously. The prompt expects a reference answer, the outputs from multiple models, and an optional consistency rubric. It produces a structured report flagging which models diverge, how they diverge, and whether the divergence is semantic, structural, or stylistic.

Do not use this prompt when you lack a reliable reference answer—this is not a reference-free quality scorer. Do not use it for single-model evaluation where pairwise comparison or rubric-based scoring would be more appropriate. Do not use it when your models are intentionally designed to produce different output styles or formats, as the consistency metric will flag intentional variation as failure. For high-stakes domains like healthcare or finance, always include human review of flagged divergences before taking action. The prompt identifies inconsistency; it does not judge which model is correct when they disagree with each other and the reference.

This prompt template is the core of the cross-model consistency playbook. It is designed to be model-agnostic, accepting the outputs from two or more models, a reference answer, and your evaluation criteria. The template instructs the LLM judge to perform a structured comparison, identifying where models diverge from the reference and from each other, and to categorize the severity and nature of those divergences. Use this as a starting point, adapting the [CONSTRAINTS] and [OUTPUT_SCHEMA] to match your specific risk tolerance and downstream logging format.

textCopy
You are an expert evaluation judge. Your task is to compare the outputs of multiple AI models against a single, authoritative reference answer. Your goal is to produce a Cross-Model Consistency Report that identifies divergence patterns and model-specific failure modes.

## INPUTS
### REFERENCE ANSWER
[REFERENCE_ANSWER]

### MODEL OUTPUTS
[OUTPUT_MODEL_A]
[OUTPUT_MODEL_B]
[OUTPUT_MODEL_C]

### EVALUATION CRITERIA
[EVALUATION_CRITERIA]

## INSTRUCTIONS
1.  **Analyze the Reference:** Extract the core facts, logical steps, and conclusions from the `[REFERENCE_ANSWER]`.
2.  **Compare Each Model Output:** For each model output, perform a claim-by-claim comparison against the reference. Classify each claim as:
    - **`match`**: Semantically identical to the reference.
    - **`contradiction`**: Directly conflicts with the reference.
    - **`hallucination`**: Introduces new information not present in the reference.
    - **`omission`**: Fails to include a key fact from the reference.
3.  **Analyze Cross-Model Divergence:** Identify where the model outputs diverge from *each other*, even if one matches the reference. Note if models make the same mistake in the same way.
4.  **Produce the Final Report:** Generate a JSON object conforming to the `[OUTPUT_SCHEMA]` below. Do not include any text outside the JSON object.

## CONSTRAINTS
[CONSTRAINTS]

## OUTPUT SCHEMA
[OUTPUT_SCHEMA]

To adapt this template, start by replacing the [REFERENCE_ANSWER] and [OUTPUT_MODEL_X] placeholders with your actual data. The [EVALUATION_CRITERIA] is where you define what 'consistent' means for your use case—this could be factual accuracy, tone, structure, or a weighted combination. The [CONSTRAINTS] placeholder is critical for high-risk domains; use it to add rules like 'If any model output contains a contradiction in a safety-critical field, the report's top-level pass field must be false.' Finally, replace [OUTPUT_SCHEMA] with a strict JSON schema that your application parser expects, ensuring the judge's output can be directly ingested by your logging and alerting systems without manual reformatting.

This prompt is designed to be called programmatically as part of a multi-model evaluation harness, not as a one-off manual check. The typical integration pattern is a Python or TypeScript script that iterates over a golden dataset of [INPUT] and [REFERENCE_OUTPUT] pairs, sends each pair to every model under test, collects the raw responses, and then feeds those responses into this consistency prompt. The prompt expects a structured input object containing the original input, the reference output, and a map of model identifiers to their respective outputs. The output is a JSON report with a consistency_score, model_rankings, divergence_patterns, and model_specific_failure_modes array. You should parse this JSON and store it alongside the run metadata for trend analysis over time.

Build the harness with three layers: input assembly, execution with retries, and output validation. For input assembly, construct the [MODEL_OUTPUTS] map by running each model against the same [INPUT] and [REFERENCE_OUTPUT] pair. Use a consistent temperature setting (0.0 is recommended for reproducibility) and capture the full raw response. For execution, call the LLM judge with this prompt and implement a retry loop: if the response fails JSON parsing, retry up to 2 times with an appended instruction to fix the parse error. For output validation, define a strict schema using Pydantic or Zod that checks for the presence of consistency_score (float 0-1), model_rankings (ordered list of model IDs), divergence_patterns (list of objects with pattern_name and affected_models), and model_specific_failure_modes (list of objects with model_id and failure_mode). If validation fails after retries, log the raw response and flag the run for human review. This is a high-signal evaluation prompt where a malformed output likely indicates a deeper model failure or prompt injection.

Log every evaluation run with the following metadata: run_id, timestamp, model_versions (map of model ID to exact version or deployment name), prompt_template_version, input_hash, and the full validated output. Store these in a queryable database or log sink so you can track consistency drift over time. When a new model version is deployed, run the full golden dataset through this harness and compare the consistency_score distribution against the previous version. A drop in mean consistency greater than 0.1 should trigger an automated alert and block the release. Avoid running this prompt on outputs that are expected to be creative or divergent by design—this prompt penalizes legitimate variation. Reserve it for tasks where output consistency is a product requirement, such as structured data extraction, classification, or safety policy enforcement.