Model Upgrade Grounding Regression Test Prompt

This prompt is designed for AI platform teams and MLOps engineers who are validating a model migration or upgrade. The core job-to-be-done is to ensure that a new model version does not introduce regressions in factual grounding when compared to the current production model. The ideal user has access to a holdout set of production traces, each containing the original user input, the retrieved context, and the output from the old model. The prompt systematically compares the new model's output against this baseline, flagging any new, unsupported claims that the old model correctly avoided.

Use this prompt when you are moving from one model version to another (e.g., gpt-4-turbo to gpt-4o, or claude-3-opus to claude-3.5-sonnet) and grounding fidelity is a critical success metric. It is appropriate for RAG applications, customer support bots, or any system where the model must synthesize answers strictly from provided evidence. The prompt requires a structured input: a set of traces with [OLD_MODEL_OUTPUT], [NEW_MODEL_OUTPUT], and the shared [RETRIEVED_CONTEXT]. It is not a general-purpose hallucination detector; it specifically isolates regressions introduced by the model change, not pre-existing issues in the retrieval pipeline.

Do not use this prompt for evaluating a single model in isolation, for real-time production guardrails, or for tasks where the output format is unstructured creative text. It is a batch evaluation tool for pre-release QA. The output is a structured regression report, not a conversational summary. If your workflow involves regulated content (e.g., medical, legal, or financial advice), the output of this prompt is a diagnostic signal, not a final compliance check. A human reviewer must still sign off on any critical grounding regressions before the new model is promoted to production. Proceed to the next section to copy the prompt template and adapt it to your specific trace schema and grounding criteria.

This prompt template is designed to be executed programmatically as part of a model upgrade validation pipeline. It takes a single trace sample—containing the original user query, the retrieved context, and the outputs from both the old and new models—and produces a structured regression verdict. The core job is to identify whether the new model introduced unsupported claims that the old model avoided, making it a critical gate before promoting a model version in a RAG system.

textCopy
You are a grounding regression analyst. Your task is to compare two model responses generated from the same user query and the same retrieved context. You must determine whether the [NEW_MODEL_NAME] response introduced any factual claims that are not supported by the provided context and that were not present in the [OLD_MODEL_NAME] response.

## INPUT

**User Query:**
[USER_QUERY]

**Retrieved Context:**
[RETRIEVED_CONTEXT]

**Old Model Response ([OLD_MODEL_NAME]):**
[OLD_MODEL_OUTPUT]

**New Model Response ([NEW_MODEL_NAME]):**
[NEW_MODEL_OUTPUT]

## INSTRUCTIONS

1.  **Extract Atomic Claims:** From the [NEW_MODEL_NAME] response, extract every discrete factual claim. A claim is a statement that can be verified against the retrieved context.
2.  **Ground Each Claim:** For each claim, determine if it is **Supported** (directly stated in the context), **Partially Supported** (context implies it but does not state it explicitly), or **Unsupported** (no evidence in the context).
3.  **Compare to Old Model:** For each claim, check if a semantically equivalent claim was present in the [OLD_MODEL_NAME] response.
4.  **Flag Regressions:** A regression is an **Unsupported** claim in the new model's output that was **not present** in the old model's output. Partially supported claims that introduce new speculative details are also regressions.
5.  **Ignore Shared Hallucinations:** If both models made the same unsupported claim, flag it as a pre-existing issue, not a regression.

## OUTPUT FORMAT

Return a single JSON object matching this schema:
{
  "verdict": "PASS" | "REGRESSION_DETECTED" | "INCONCLUSIVE",
  "summary": "A one-sentence summary of the finding.",
  "new_model_claims": [
    {
      "claim_text": "The exact claim from the new model.",
      "grounding_status": "Supported" | "Partially Supported" | "Unsupported",
      "closest_context_quote": "The most relevant sentence from the retrieved context, or null.",
      "present_in_old_model": true | false,
      "is_regression": true | false
    }
  ],
  "regression_details": [
    {
      "unsupported_claim": "The regressed claim text.",
      "reason": "Why this claim is unsupported and new."
    }
  ]
}

## CONSTRAINTS

- Do not penalize the new model for better writing style, grammar, or fluency.
- A claim is only a regression if it introduces new, unsupported factual content.
- If the retrieved context is empty or irrelevant, state that grounding is impossible and set the verdict to "INCONCLUSIVE".
- If the new model correctly abstained or expressed uncertainty where the old model did not, that is not a regression.

To adapt this template, replace the square-bracket placeholders with data from your trace store. For a batch regression run, iterate over a holdout set of traces, populating [USER_QUERY], [RETRIEVED_CONTEXT], [OLD_MODEL_OUTPUT], and [NEW_MODEL_OUTPUT] for each sample. The [OLD_MODEL_NAME] and [NEW_MODEL_NAME] placeholders should be set to the specific model identifiers (e.g., gpt-4-0613 and gpt-4-1106-preview) to help the evaluator model maintain context. For high-stakes migrations, ensure a human reviews all REGRESSION_DETECTED verdicts before blocking the upgrade, and log the full prompt and response for auditability.

This prompt is designed to run as a batch evaluation step, not as a real-time user-facing call. It compares outputs from an old model version and a new model version against the same retrieved context captured in a holdout trace set. The implementation harness must feed both models the identical prompt, context, and generation parameters, then route the paired outputs into this evaluation prompt for structured comparison. The harness is responsible for ensuring that any differences in grounding are attributable to the model change, not to variations in retrieval, temperature, or system instructions.

The harness should iterate over a golden trace dataset where each record contains: the original user query, the retrieved context (documents, chunks, or passages), the old model's generated response, and the new model's generated response. For each record, inject these values into the prompt's [OLD_MODEL_OUTPUT], [NEW_MODEL_OUTPUT], and [RETRIEVED_CONTEXT] placeholders. Set [EVALUATION_CRITERIA] to a structured rubric that flags unsupported claims, citation mismatches, and factual additions. Run the evaluation with a low-temperature setting (0.0–0.2) to maximize consistency. Capture the structured output—ideally JSON with fields for regression_detected, unsupported_claims_added, supported_claims_removed, and severity—and write it to a results store for aggregation. Implement a retry wrapper: if the output fails to parse as valid JSON or is missing required fields, retry up to two times with a repair prompt before logging a parse failure and marking the record for human review.

For production-grade reliability, add a validation layer that checks the evaluation output against a set of invariants. For example, if regression_detected is true but unsupported_claims_added is empty, flag the record as inconsistent. Log every evaluation result with the trace ID, model version pair, timestamp, and raw evaluation output. This log becomes the audit trail for the model migration decision. Before promoting the new model, aggregate results across the full holdout set and apply a statistical threshold: if the regression rate exceeds a predefined tolerance (e.g., more than 2% of traces show new unsupported claims), block the migration and surface the worst-affected trace IDs for manual diagnosis. Do not rely on a single pass/fail from one trace—this prompt is a building block in a broader regression gate, not the gate itself.