Prompt Change Impact Analysis Prompt

This prompt is for prompt engineers and release managers who need a structured pre-flight risk assessment before modifying a production prompt. Instead of running the full regression suite blindly or shipping and hoping, you provide a concrete prompt diff and a versioned golden dataset, and the model predicts which test cases are likely to break, estimates the blast radius, assigns confidence scores to each prediction, and suggests mitigations. The goal is to make an informed decision about whether the change is safe enough to proceed to full regression testing or whether it needs rework first.

The prompt works best when your golden dataset includes diverse input categories, difficulty tags, and expected behaviors so the model can reason about change impact rather than guessing. You should provide the exact diff between the current and proposed prompt, not just a summary of the change. The model needs to see the specific wording changes, added constraints, removed instructions, or reordered sections to assess impact accurately. Include metadata about each test case—such as category, difficulty, and expected output shape—so the model can correlate change types with likely failure modes. The output is a structured risk report with per-case predictions, confidence scores, blast radius estimates, and suggested mitigations like targeted re-testing, human review, or prompt rollback.

Do not use this prompt as a replacement for actual regression testing. It is a pre-flight risk analysis tool, not a test executor. The model can miss subtle interactions between prompt changes and edge cases, and its confidence scores are estimates, not guarantees. Always run the full regression suite before deploying to production, and use this prompt to decide whether the change is worth the testing cost or needs revision first. If your golden dataset is sparse, outdated, or lacks diversity, the risk predictions will be unreliable. Invest in dataset quality before relying on this workflow.

This prompt template is designed to be pasted directly into your evaluation harness. It instructs the model to analyze a proposed prompt diff against a golden dataset and produce a structured JSON risk assessment. The model acts as a regression impact analyst, not a general-purpose assistant. It must identify which test cases are likely to break, estimate the blast radius, assign confidence scores, and suggest mitigations. The prompt uses square-bracket placeholders that you replace with your actual data before execution. Do not modify the core instruction structure unless you also update your evaluation criteria to match.

textCopy
You are a regression impact analyst. Your job is to assess the risk of a proposed prompt change before it ships.

You will receive:
- A current prompt template: [CURRENT_PROMPT]
- A proposed prompt template with changes: [PROPOSED_PROMPT]
- A golden dataset of test cases: [GOLDEN_DATASET]

For each test case in the golden dataset, predict whether the proposed change will cause a regression. A regression means the output for that test case would become incorrect, malformed, or violate the output contract in a way that the current prompt does not.

Output a JSON object with this exact schema:
{
  "analysis": {
    "summary": "string: one-paragraph summary of overall risk",
    "blast_radius_estimate": "string: low | medium | high | critical",
    "total_test_cases": "integer",
    "predicted_regressions": "integer",
    "predicted_improvements": "integer",
    "unchanged_predictions": "integer"
  },
  "test_case_predictions": [
    {
      "test_case_id": "string",
      "prediction": "string: regression | improvement | unchanged",
      "confidence": "float between 0.0 and 1.0",
      "rationale": "string: why this test case is predicted to change",
      "suggested_mitigation": "string or null: how to reduce risk for this case"
    }
  ],
  "high_risk_cases": ["array of test_case_id strings with confidence below 0.7 or predicted regression"],
  "mitigation_recommendations": ["array of strings: global mitigations to apply before shipping"]
}

Constraints:
- Do not fabricate test cases. Only analyze the provided golden dataset.
- If the diff is unclear or ambiguous, flag it in the summary and assign lower confidence.
- For each predicted regression, explain what specific change in the prompt caused the prediction.
- If a test case input is missing required context after the change, flag it as a regression.
- Do not recommend shipping without mitigations when blast_radius_estimate is high or critical.

To adapt this template, replace each square-bracket placeholder with your actual data. [CURRENT_PROMPT] and [PROPOSED_PROMPT] should contain the full prompt text, not just the diff. [GOLDEN_DATASET] should be a structured list of test cases, each with an id, input, and expected_output field at minimum. If your golden dataset includes metadata like difficulty tags or category labels, include those fields so the model can use them in its rationale. Before running this prompt in a CI/CD pipeline, validate that the output JSON matches the schema exactly. A single missing field or type mismatch will break downstream automation. Consider adding a schema validation step after the model response and retrying once if validation fails. For high-risk prompt changes where blast_radius_estimate is critical, route the output to a human reviewer before accepting the risk assessment.

The Prompt Change Impact Analysis Prompt is designed to run before a prompt change reaches production, not as a post-deployment audit. It takes a proposed prompt diff and a golden dataset as inputs, then predicts which test cases are likely to break and estimates the blast radius. The harness must ensure that the prompt diff is complete and unambiguous, the golden dataset is version-locked, and the analysis output is structured enough to feed into a release gate decision. This prompt is a risk assessment tool, not a regression test runner—it predicts impact, it does not execute the test suite.

Wire this prompt into a pre-commit or pull-request CI step for your prompt repository. The harness should: (1) extract the proposed prompt diff from the PR, (2) retrieve the current golden dataset from a versioned artifact store, (3) format both into the [PROMPT_DIFF] and [GOLDEN_DATASET] placeholders, (4) call the prompt with a deterministic model configuration (temperature=0, seed fixed), and (5) parse the JSON output into a structured risk report. Validate the output schema strictly: every predicted breakage must include a test_case_id, confidence_score between 0.0 and 1.0, and a failure_mode category. Reject malformed outputs and retry once with a repair prompt before failing the CI step. Log the full prompt, response, and validation result for auditability.

For high-risk prompt changes—those affecting safety policies, financial calculations, or regulated outputs—add a human approval gate after the impact analysis runs. The harness should block the merge if the predicted blast radius exceeds a configurable threshold (e.g., more than 5% of test cases predicted to break with confidence above 0.7). Store the analysis alongside the prompt version in your prompt registry so that post-deployment debugging can compare predicted impact against actual regression test results. Avoid running this prompt on every minor wording change; reserve it for structural, behavioral, or constraint-level modifications where the risk of silent breakage is real.