Model Upgrade Impact Assessment Prompt

This prompt is designed for AI platform engineers and release managers who need to evaluate the blast radius of a foundation model upgrade before migrating production traffic. The job-to-be-done is producing a structured impact report that compares prompt behavior across model versions, identifies regressions, estimates remediation effort, and provides clear rollback decision criteria. It is not a general-purpose model comparison tool, a benchmark runner, or a prompt optimizer. Use it when you have a specific prompt or set of prompts running on a current model version and you are planning a migration to a newer version (e.g., GPT-4 to GPT-4o, Claude 3 to Claude 3.5, or an open-weight model upgrade).

The ideal user is someone who owns the prompt's production behavior and can supply the current prompt template, a golden test suite of representative inputs with expected outputs, the current model identifier, and the target model identifier. Required context includes the prompt's role in the application, any output contracts (schemas, tool calls, formatting rules), and the risk tolerance of the downstream system. Do not use this prompt when you lack a golden dataset, when the prompt is purely conversational with no measurable output contract, or when you are comparing models for initial selection rather than assessing an upgrade's impact on an existing, stable prompt. The prompt assumes you have already built a regression test suite; it does not create one from scratch.

This prompt is high-stakes by nature: a bad model migration can silently degrade user-facing behavior, break structured outputs, or introduce subtle semantic drift that evades simple string matching. The output must be reviewed by a human before any migration decision is finalized. The prompt is designed to surface evidence, not to make the final call. After running this assessment, you should feed the results into your release gate process, document any accepted regressions with justification, and ensure rollback triggers are configured before cutting over traffic.

This section provides the core prompt you will use to compare prompt behavior across model versions. The template is designed to be wired into an automated evaluation harness, but it can also be run manually during pre-migration research. It forces the model to act as an evaluator, not the system under test, and produces a structured report that downstream tooling can parse.

textCopy
You are an AI platform engineer evaluating the impact of a foundation model upgrade.

Your task is to compare the behavior of a single prompt template across two model versions and produce a structured impact assessment report.

# INPUTS

[PROMPT_UNDER_TEST]

[INPUT_EXAMPLES]

[OUTPUTS_FROM_CURRENT_MODEL]

[OUTPUTS_FROM_CANDIDATE_MODEL]

[EVALUATION_CRITERIA]

[REGRESSION_THRESHOLDS]

# INSTRUCTIONS

1. For each input example, compare the output from the current model against the output from the candidate model using the provided evaluation criteria.
2. Classify each comparison as PASS, REGRESSION, IMPROVEMENT, or SEMANTIC_DRIFT.
3. For any REGRESSION or SEMANTIC_DRIFT, estimate the remediation effort as LOW, MEDIUM, HIGH, or BLOCKER.
4. Aggregate results into the output schema below.
5. If the candidate model introduces a BLOCKER regression on any example rated CRITICAL in [REGRESSION_THRESHOLDS], set the overall migration recommendation to ROLLBACK.

# OUTPUT SCHEMA

Return a valid JSON object with this exact structure:
{
  "assessment_id": "string",
  "current_model": "string",
  "candidate_model": "string",
  "prompt_under_test": "string",
  "summary": {
    "total_examples": number,
    "pass_count": number,
    "regression_count": number,
    "improvement_count": number,
    "semantic_drift_count": number,
    "blocker_count": number
  },
  "per_example_results": [
    {
      "example_id": "string",
      "classification": "PASS | REGRESSION | IMPROVEMENT | SEMANTIC_DRIFT",
      "severity": "CRITICAL | HIGH | MEDIUM | LOW",
      "remediation_effort": "LOW | MEDIUM | HIGH | BLOCKER",
      "current_output_summary": "string",
      "candidate_output_summary": "string",
      "difference_description": "string",
      "affected_evaluation_criteria": ["string"]
    }
  ],
  "migration_recommendation": "PROCEED | PROCEED_WITH_CAUTION | ROLLBACK",
  "rollback_triggers_met": ["string"],
  "estimated_remediation_effort_total": "LOW | MEDIUM | HIGH | BLOCKER",
  "notes": "string"
}

# CONSTRAINTS

- Do not hallucinate differences. Only report differences you can observe in the provided outputs.
- If an output is missing or malformed, classify it as a REGRESSION with severity HIGH.
- Do not propose prompt changes. This assessment is for impact visibility only.
- If [REGRESSION_THRESHOLDS] specifies that any CRITICAL regression is a blocker, you must recommend ROLLBACK.

To adapt this template, replace each square-bracket placeholder with concrete data from your test harness. [PROMPT_UNDER_TEST] should contain the exact system and user prompt you are evaluating. [INPUT_EXAMPLES] should be a JSON array of test inputs drawn from your golden dataset. [OUTPUTS_FROM_CURRENT_MODEL] and [OUTPUTS_FROM_CANDIDATE_MODEL] should contain the raw model responses for each input, keyed by example ID. [EVALUATION_CRITERIA] should list the dimensions you care about, such as factual accuracy, schema validity, tone, or citation compliance. [REGRESSION_THRESHOLDS] should define which example severities and criteria failures trigger a ROLLBACK recommendation. After running this prompt, validate the output JSON against the schema before feeding it into your release gate decision logic. For high-risk domains, always include a human review step before accepting a PROCEED recommendation.

This prompt is designed to be the decision-support layer in a larger automated regression testing harness, not a one-off chat interaction. The harness should execute the prompt programmatically against a golden dataset of representative inputs, collecting outputs from both the current production model and the candidate upgrade model. The prompt's [OUTPUT_SCHEMA] placeholder should be populated with a strict JSON schema that your test runner can parse and validate automatically. A typical integration involves a Python or TypeScript script that iterates over your golden examples, calls the LLM API for each model version, parses the structured JSON response, and writes the results to a structured log or database for analysis.

The harness must include validation and retry logic to ensure the prompt's output is machine-readable. After receiving the model's response, validate it against the expected JSON schema. If parsing fails, implement a retry loop (maximum 2-3 attempts) that feeds the raw output and the validation error back to the model with a correction instruction. Log every retry event. For high-stakes migrations, the harness should also enforce a human review gate: if the [RISK_LEVEL] is set to high or if the model flags any critical regressions in the remediation_effort field, the pipeline should halt and create a review ticket rather than automatically proceeding. Model choice is critical here; use a model with strong instruction-following and structured output capabilities (such as GPT-4o or Claude 3.5 Sonnet) to generate the assessment, as smaller or older models may fail to adhere to the complex output schema reliably.

To operationalize this, wire the prompt into a CI/CD pipeline that triggers on a new model version availability or a scheduled cadence. The harness should produce an artifact—a machine-readable impact report—that downstream release gates can consume. Avoid the temptation to treat this prompt as a final approval; it is a diagnostic tool. The rollback_decision_criteria field it generates should be treated as a recommendation that your release script evaluates against hard metric thresholds (e.g., 'rollback if >5% of tests show semantic drift'). Always log the full prompt payload, model response, and validation status for auditability, and never allow an unvalidated model upgrade to reach production based solely on this prompt's output.