Model Migration Factual Regression Prompt

Use this prompt when you are migrating an AI-powered product feature from one model to another—for example, moving from GPT-4 to GPT-4o, from Claude 3 Opus to Claude 3.5 Sonnet, or from a proprietary model to an open-weight alternative—and you need to verify that factual accuracy does not regress. The core job is to compare outputs from the old model and the new model on a shared evaluation set, identify cases where the new model introduces hallucinations or unsupported claims that the old model did not, and produce a prioritized list of regression cases with actionable fix recommendations. The ideal user is an AI engineer, MLOps engineer, or technical product lead who owns model migration and needs evidence before cutting over production traffic.

This prompt is designed for structured, repeatable evaluation workflows. You should provide a set of test inputs, the corresponding outputs from the old model (the baseline), the outputs from the new model (the candidate), and the source context or ground-truth evidence used to generate those outputs. The prompt works best when each test case includes the original user query, any retrieved documents or system context, and both model responses side by side. It assumes you have already run both models against the same inputs and collected their outputs—this prompt does not call the models itself. It is a post-generation analysis and comparison tool, not a live routing or retry mechanism.

Do not use this prompt when you are evaluating a single model in isolation, when you lack a baseline from a previous model version, or when your evaluation set is too small to produce meaningful regression signals (fewer than 20-30 diverse test cases is usually insufficient). This prompt is also not a substitute for automated factual accuracy scoring pipelines or LLM-judge eval harnesses—it complements those by adding comparative regression analysis. For high-risk domains such as healthcare, legal, or finance, always include human review of flagged regressions before accepting the prompt's recommendations. The output should feed into your prompt adaptation process, not directly into production without validation.

This prompt template is designed to be dropped into an evaluation harness that runs the same input set against an old model version and a new model version, then compares the outputs for factual regression. The template expects a shared evaluation input, the outputs from both models, and the source context that both models should have used. It produces a structured regression report that flags cases where the new model introduced hallucinations, degraded citation accuracy, or fabricated details that the old model handled correctly.

codeCopy
You are a factual regression auditor comparing outputs from two model versions against verified source context.

## INPUT
Old Model Output: [OLD_MODEL_OUTPUT]
New Model Output: [NEW_MODEL_OUTPUT]
Source Context: [SOURCE_CONTEXT]
Evaluation Input: [EVALUATION_INPUT]

## TASK
Compare the factual accuracy of the two outputs against the source context. Identify cases where the new model output is factually worse than the old model output. A regression occurs when:
- The new model introduces a claim not present in the source context that the old model did not make.
- The new model fabricates a specific detail (number, date, name, entity) that the old model got correct or omitted.
- The new model misattributes a source or cites a passage that does not support its claim, where the old model cited correctly.
- The new model contradicts the source context on a point where the old model was accurate.
- The new model makes an internally inconsistent statement that the old model did not make.

Do NOT flag cases where both models made the same error, or where the new model is more accurate than the old model. This audit is specifically for regressions.

## OUTPUT SCHEMA
Return a JSON object with this structure:
{
  "regression_detected": boolean,
  "regression_count": number,
  "regressions": [
    {
      "regression_id": "string",
      "severity": "critical|high|medium|low",
      "regression_type": "fabrication|misattribution|contradiction|inconsistency|detail_error",
      "old_model_claim": "string (what the old model said)",
      "new_model_claim": "string (what the new model said)",
      "source_evidence": "string (the relevant source passage)",
      "explanation": "string (why this is a regression)",
      "fix_suggestion": "string (how to adjust the prompt or constraints)"
    }
  ],
  "summary": "string (one-paragraph summary of regression patterns)",
  "prompt_fix_recommendations": ["string (actionable prompt changes)"]
}

## CONSTRAINTS
- Only flag regressions where the new model is worse than the old model.
- Every flagged regression must include the exact source evidence that contradicts the new model.
- If no regressions are found, return regression_detected: false with an empty regressions array.
- Do not evaluate style, tone, or verbosity. Only factual accuracy against source context.
- If the source context is insufficient to verify a claim, note it but do not flag it as a regression unless the old model correctly abstained and the new model did not.

Adapt this template by adjusting the regression criteria to match your domain's factual accuracy requirements. For regulated industries, add domain-specific fact patterns to the regression types list. For RAG systems, tighten the citation verification criteria. For agent workflows, extend the output schema to include the tool call or action step where the regression occurred. Always pair this prompt with a validation step that checks the output JSON against the expected schema before feeding results into your regression dashboard or CI pipeline.

This prompt is designed to run inside an automated evaluation pipeline, not as a one-off manual check. The harness must feed the prompt a structured evaluation record containing the original query, the retrieved or provided context, and the outputs from both the old and new models. Because the prompt performs comparative factual analysis, the harness is responsible for ensuring that both model outputs were generated under identical conditions—same system prompt, same context, same tool access—so that any regression is attributable to the model change rather than environmental drift.

Wire the prompt into a batch evaluation runner that iterates over a golden evaluation set. Each record should include [QUERY], [CONTEXT], [OLD_MODEL_OUTPUT], [NEW_MODEL_OUTPUT], and [EVAL_ID]. The harness must validate the prompt's structured output against a strict schema: a JSON object containing a regression_detected boolean, a severity enum (none, low, medium, high, critical), an array of regression_cases with claim, old_model_support, new_model_issue, and evidence fields, and a fix_recommendations array. Use a JSON schema validator in the harness to reject malformed responses and trigger a single retry with the validation error appended. Log every evaluation result—including parse failures—with the [EVAL_ID] for traceability.

For high-stakes migrations, insert a human review gate for critical severity regressions before accepting the prompt's fix recommendations. The harness should aggregate results into a migration readiness report: total cases evaluated, regression rate by severity, and a prioritized list of prompt adaptations. Avoid running this prompt on streaming outputs; the comparative analysis requires complete responses from both models. If the new model introduces a different output structure, pre-process both outputs into a normalized claim format before invoking the regression prompt to prevent false positives from format differences rather than factual errors.