Model Behavior Change Regression Prompt

This prompt is designed for ML engineers and AI reliability engineers who need to determine whether a regression in evaluation scores is caused by prompt brittleness or a genuine degradation in model capability. It assumes you have access to pre-change and post-change eval results, a stable golden dataset, and production trace data for the same inputs. The job-to-be-done is differential root-cause analysis: isolating which behavior dimensions changed, quantifying the shift, and attributing it to either the prompt's sensitivity to the new model or a real capability regression that requires model-level intervention.

Before using this prompt, you must assemble a structured comparison dataset. This includes a set of identical inputs run through both the old and new model configurations, their corresponding outputs, and the evaluation scores broken down by dimension (e.g., factual accuracy, instruction following, format compliance, safety refusals). The prompt works best when you can provide per-dimension score distributions, not just aggregate metrics. A 2% drop in overall accuracy is unactionable; a 15% drop in citation precision alongside stable factual accuracy points directly to a retrieval or grounding behavior change. The prompt's differential analysis template forces this dimensional breakdown and will flag when your input data is too coarse to support a conclusion.

Do not use this prompt for initial prompt design, for debugging a single bad output, or when you lack a stable golden dataset for comparison. For isolated incident investigation, use the Eval Failure Root-Cause Triage Prompt instead. After running this analysis, expect to take one of three paths: adjust the prompt to accommodate the new model's behavior, roll back the model change if the regression is severe and prompt-agnostic, or accept the shift and update your eval thresholds to reflect the new baseline. The prompt's output includes a recommended action and a confidence level, but the final decision should involve human review of the evidence, especially when the regression affects safety-critical or compliance-sensitive dimensions.

This prompt is designed to be pasted directly into an automated analysis pipeline or an investigation notebook. It accepts two sets of evaluation results—one from before a model change and one from after—along with the prompt template used and any relevant trace samples. The model is instructed to produce a structured differential analysis, not a generic summary. Replace every square-bracket placeholder with your actual data before running the prompt. The output is a JSON report that can be ingested by downstream monitoring dashboards or regression tracking systems.

textCopy
You are an ML reliability engineer performing a root-cause analysis of an eval score shift after a model provider update or internal model swap.

You are given:
- [PRE_CHANGE_EVAL_RESULTS]: Evaluation results from the baseline model version, including per-dimension scores, failure counts, and sample failures.
- [POST_CHANGE_EVAL_RESULTS]: Evaluation results from the new model version, structured identically to the pre-change results.
- [PROMPT_TEMPLATE]: The exact prompt template used for both runs, with placeholders preserved.
- [TRACE_SAMPLES]: A representative set of production traces from both periods, including retrieved context, tool calls, and final outputs. Include at least 5 traces where the eval score changed significantly.
- [EVAL_RUBRIC]: The evaluation rubric defining each scored dimension, its weight, and pass/fail criteria.
- [CHANGE_DESCRIPTION]: A brief description of what changed (e.g., model version, provider, deployment date).

Produce a JSON report with the following structure:
{
  "summary": {
    "overall_score_delta": float,
    "significant_dimensions": ["dimension_name"],
    "primary_root_cause_category": "PROMPT_BRITTLENESS" | "MODEL_CAPABILITY_REGRESSION" | "DATA_DRIFT" | "INCONCLUSIVE"
  },
  "dimension_analysis": [
    {
      "dimension": "string",
      "score_delta": float,
      "direction": "IMPROVED" | "DEGRADED" | "UNCHANGED",
      "likely_cause": "PROMPT_BRITTLENESS" | "MODEL_CAPABILITY_REGRESSION" | "DATA_DRIFT",
      "evidence": ["trace_id or specific observation"],
      "confidence": "HIGH" | "MEDIUM" | "LOW"
    }
  ],
  "brittleness_indicators": [
    {
      "indicator": "string describing a prompt brittleness signal",
      "observed_in_dimensions": ["dimension_name"],
      "severity": "HIGH" | "MEDIUM" | "LOW"
    }
  ],
  "capability_regression_indicators": [
    {
      "indicator": "string describing a genuine capability regression signal",
      "observed_in_dimensions": ["dimension_name"],
      "severity": "HIGH" | "MEDIUM" | "LOW"
    }
  ],
  "recommendations": [
    {
      "action": "string describing a concrete next step",
      "target": "PROMPT" | "MODEL" | "EVAL" | "DATA",
      "priority": "HIGH" | "MEDIUM" | "LOW",
      "rationale": "string"
    }
  ],
  "requires_human_review": true | false,
  "human_review_focus": ["specific areas needing human judgment"]
}

Rules:
- Compare pre- and post-change results dimension by dimension.
- For each dimension with a significant delta (absolute change > [SIGNIFICANCE_THRESHOLD]), determine whether the shift is consistent with prompt brittleness (e.g., the model now interprets an instruction differently, few-shot examples mislead, output format constraints are violated) or genuine capability regression (e.g., the model is worse at reasoning, summarization, or instruction following independent of prompt wording).
- Use the trace samples to ground your analysis. Cite specific trace IDs or patterns.
- If the evidence is insufficient to determine root cause, set primary_root_cause_category to "INCONCLUSIVE" and flag requires_human_review as true.
- Do not fabricate evidence. If a trace does not support a conclusion, state that explicitly.
- If the eval rubric itself may be brittle (e.g., it penalizes valid paraphrasing), include that in the recommendations with target "EVAL".

To adapt this prompt for your pipeline, start by setting [SIGNIFICANCE_THRESHOLD] to a value that reflects your team's tolerance for false positives—0.05 is a common starting point for normalized scores. The [TRACE_SAMPLES] field is critical: without representative traces from both periods, the model cannot ground its analysis in evidence. If your observability platform captures traces with step-level attribution, include the full trace object. If not, include at minimum the user input, retrieved context, final output, and eval score for each sampled request. The output JSON is designed to be machine-readable; pipe it directly into your regression tracking database or incident response channel. For high-risk domains where a model behavior change could affect safety or compliance, always set requires_human_review to true in your post-processing logic if the model's confidence is LOW on any degraded dimension.

This prompt is designed to be the analytical core of a semi-automated regression investigation pipeline, not a one-off chat interaction. The harness must feed it structured input from your eval store and model gateway logs, then parse its differential analysis for downstream actions. The primary integration points are your evaluation results database (for pre- and post-change score distributions), your prompt version control system (for the exact system prompt and few-shot examples used), and your model gateway's trace logs (for sample outputs from both the baseline and candidate model versions).

The implementation should follow a validate-extract-route pattern. First, validate that the required inputs—[BASELINE_EVAL_RESULTS], [CANDIDATE_EVAL_RESULTS], [PROMPT_TEMPLATE], and [SAMPLE_OUTPUTS]—are present and correctly formatted. A pre-processing script should sample a representative set of failing cases from the candidate eval run, ensuring coverage across all degraded eval dimensions. Second, after the LLM returns its analysis, extract the structured differential_analysis JSON block. Use a strict JSON schema validator to confirm the presence of required fields like behavior_dimensions, root_cause_classification, and recommended_actions. If parsing fails, retry the prompt once with a stronger format instruction; if it fails again, log the raw output and escalate for human review. Third, route the validated output: if root_cause_classification is prompt_brittleness, open a ticket in the prompt engineering backlog with the identified fragile instructions. If it's genuine_model_regression, route the report to the model evaluation lead and flag the model version in your gateway for potential rollback or A/B testing.

For high-stakes production systems, never allow this prompt's output to automatically trigger a prompt rollback or model routing change. The harness must enforce a human-in-the-loop gate before any remediation action. Log every run of this prompt as an investigation artifact, including the full input payload, the raw LLM response, and the parsed analysis. This creates an audit trail for post-mortems and helps you detect if the LLM judge itself begins to drift in its classification behavior over time. Start by running this prompt on a small, known regression incident to calibrate your validation thresholds and routing rules before scaling it to automated monitoring triggers.