Model Upgrade Regression Test Prompt

This prompt is designed for prompt engineering teams migrating existing prompts to a new model version. When you upgrade from one model checkpoint to another, the same prompt text can produce different outputs due to changes in model behavior, tokenization, instruction following, or output formatting. This prompt runs your existing prompt and golden test set against the target model, compares outputs systematically, identifies behavioral regressions specific to the model change, and flags examples that need rewriting for the new model. Use this before promoting a model upgrade to production.

The ideal user is an AI engineer or prompt author who already has a stable prompt that works on a source model and a golden test set with expected outputs. You must provide the original prompt text, the golden test cases with expected outputs, and access to the target model. The prompt will produce a structured regression report that isolates failures caused by the model change, not by prompt edits. This separation is critical: it tells you whether the prompt itself needs rewriting for the new model or whether the model's behavior has shifted in ways that require prompt adaptation.

Do not use this for ordinary prompt version testing against the same model. That belongs in the Regression Test Harness Prompt for Prompt Changes playbook. Do not use this when you lack a golden test set with verified expected outputs—the comparison logic depends on having a ground truth to measure against. Do not use this for initial prompt development or for comparing two different prompts on the same model. This prompt answers one specific question: 'When I move my existing prompt to a new model, what breaks?' If you need to build a golden test set first, start with the Golden Test Set Construction Prompt Template playbook.

The following prompt template is designed to be executed as part of an automated evaluation harness. It takes a single golden test case—consisting of a known input and its expected output—and runs it against a target model using your existing production prompt. The model's job is to compare the new output against the golden expected output and flag any behavioral regressions specific to the model change. Replace each square-bracket placeholder with concrete values before sending this to the target model. This prompt is not a chat interface; it is a machine-to-machine evaluation step that should be called programmatically.

textCopy
You are an AI regression analyst. Your task is to compare a model output generated by a new model version against a golden expected output and determine whether a behavioral regression has occurred.

## INPUT DATA

**ORIGINAL PROMPT (the prompt under test):**

[ORIGINAL_PROMPT]

codeCopy

**USER INPUT (the test case input):**

[TEST_CASE_INPUT]

codeCopy

**GOLDEN EXPECTED OUTPUT (from the previous model version):**

[GOLDEN_EXPECTED_OUTPUT]

codeCopy

**NEW MODEL OUTPUT (from the target model):**

[NEW_MODEL_OUTPUT]

codeCopy

## EVALUATION CRITERIA

Analyze the new model output against the golden expected output using these dimensions:

1. **Schema Compliance**: Does the new output conform to the expected output schema [OUTPUT_SCHEMA]? Check for missing required fields, incorrect types, extra fields, and enum violations.

2. **Semantic Equivalence**: Is the factual content and meaning of the new output equivalent to the golden output? Allow for acceptable paraphrasing and synonym variation. Flag substantive factual differences, omissions, or additions.

3. **Behavioral Consistency**: Does the new output exhibit the same behavioral patterns? Check for:
   - Refusal behavior: Did the model refuse when it should have answered, or answer when it should have refused?
   - Uncertainty expression: Does the model express appropriate confidence levels?
   - Tone and style: Is the tone consistent with the expected output style [STYLE_GUIDELINES]?
   - Safety boundaries: Does the output respect the same safety constraints [SAFETY_POLICIES]?

4. **Format and Structure**: Does the output follow the expected formatting conventions, including markdown, code blocks, lists, and whitespace where specified?

5. **Tool Use and Function Calls**: If the expected output includes tool calls or function invocations, does the new output select the correct tools with valid arguments?

## TOLERANCE THRESHOLDS

Apply these tolerance thresholds when evaluating differences:
- [TOLERANCE_THRESHOLDS]

## OUTPUT FORMAT

Return a JSON object with this exact structure:

```json
{
  "regression_detected": boolean,
  "severity": "none" | "low" | "medium" | "high" | "critical",
  "dimension_results": {
    "schema_compliance": {
      "passed": boolean,
      "violations": ["string description of each violation"]
    },
    "semantic_equivalence": {
      "passed": boolean,
      "differences": ["string description of each semantic difference"]
    },
    "behavioral_consistency": {
      "passed": boolean,
      "issues": ["string description of each behavioral issue"]
    },
    "format_structure": {
      "passed": boolean,
      "issues": ["string description of each format issue"]
    },
    "tool_use": {
      "passed": boolean,
      "issues": ["string description of each tool-use issue"]
    }
  },
  "regression_summary": "A concise explanation of what regressed and why, or confirmation that no regression was detected.",
  "requires_human_review": boolean,
  "suggested_action": "approve" | "review" | "reject" | "rewrite_example" | "investigate_model_behavior",
  "rewrite_suggestion": "If the example itself needs updating for the new model, provide a revised expected output. Otherwise null."
}

CONSTRAINTS

• Do not hallucinate violations. Only flag differences you can cite from the provided outputs.
• If the new output is better than the golden output, do not flag it as a regression. Note it in the summary but set regression_detected to false.
• If the golden expected output is ambiguous or incomplete, set requires_human_review to true and explain why.
• For high-severity regressions, include specific evidence from the outputs in your violation descriptions.
• If [RISK_LEVEL] is "high", always set requires_human_review to true for any detected regression.

Adapting the template: The [TOLERANCE_THRESHOLDS] placeholder is critical for tuning false-positive rates. For structured outputs, define field-level tolerances such as "numeric fields within 0.01" or "string fields must match exactly." For natural language outputs, specify semantic match thresholds like "allow paraphrasing but flag factual contradictions." The [RISK_LEVEL] placeholder controls the human-review gating behavior—set it to "high" for regulated domains, financial outputs, or clinical content. The [OUTPUT_SCHEMA] placeholder should contain your expected JSON schema, XML structure, or format specification. If your prompt does not use tools, the tool-use dimension will automatically pass. Remove or collapse dimensions that don't apply to your use case, but keep the output JSON structure stable so your eval harness can parse results consistently across test runs.

What to do next: Wire this prompt into your regression test harness so it runs automatically for every test case in your golden set. Aggregate the per-case results to produce a model upgrade pass/fail report. For any case where requires_human_review is true or severity is "high" or "critical", route the output to a review queue before approving the model migration. If multiple test cases show the same regression pattern, investigate whether the prompt itself needs rewriting for the new model rather than treating each case as an isolated failure.

This prompt is designed to be the evaluation step in a model migration pipeline, not a one-off chat interaction. The core workflow is: load a golden test set of [INPUT] and [EXPECTED_OUTPUT] pairs, run each input through the same prompt template on both the current and target model versions, then feed the paired outputs into this regression test prompt for analysis. The prompt expects structured input—a JSON array of test cases—and returns a structured regression report. Do not use this prompt for ad-hoc model comparisons; it is built for systematic, example-based regression detection where every test case has a known expected behavior.

Integration pattern: Wrap this prompt in a script or CI step that executes the following sequence. First, load your golden test set from a version-controlled file (e.g., golden_tests_v2.json). Second, for each test case, call the current model and the target model with the identical prompt template, capturing both outputs with timestamps and model version identifiers. Third, assemble the results into the [TEST_RESULTS] array required by this prompt, where each entry contains the input, expected output, current model output, and target model output. Fourth, call this regression test prompt with the assembled results. Fifth, parse the structured output—the regression_flags array and pass_fail_summary—to determine whether the migration gate passes. A CI system should block deployment if critical_regressions is non-empty or if the overall_pass field is false. Log every regression report with the model version pair for audit trails.

Validation and retries: Before calling this prompt, validate that your test results array is well-formed—every entry must have non-null values for all four required fields. If the target model returns a malformed output (e.g., invalid JSON when a schema is expected), capture the raw response and mark it as a structural failure before feeding it into this prompt. The regression test prompt itself may flag these as structural_regression types. Do not retry the regression test prompt on failure; instead, fix the underlying output collection and re-run. For high-stakes migrations (healthcare, finance, safety-critical systems), add a human review step after the automated regression report: a domain expert should inspect all critical_regression entries and a sample of moderate_regression entries before signing off on the model upgrade.

Model choice and tool use: This prompt works best with models that have strong instruction-following and structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid running regression analysis on a model weaker than the ones being compared, as analysis quality will degrade. The prompt does not require external tools or RAG; it operates entirely on the provided test results. If your golden test set exceeds ~50 examples, split it into batches of 25-40 cases to stay within context windows and avoid attention degradation on later examples. Run each batch independently and merge the regression reports. For CI/CD integration, store the prompt template itself in version control alongside the golden test set, and pin the model version used for regression analysis to ensure consistent evaluation behavior over time.

What to avoid: Do not skip the step of running the current model baseline—regressions can only be detected relative to known behavior. Do not use this prompt to compare models on tasks where the expected output is subjective or unlabeled; it requires concrete expected outputs per test case. Do not treat a passing regression report as proof that the new model is better—only that it hasn't broken known behaviors. Always supplement regression testing with net-new behavior testing, latency measurements, and cost analysis before finalizing a model upgrade decision.