Eval Score Regression Alert Prompt Template

This prompt is designed for AI product teams and quality engineers who need to detect when production eval scores have regressed below an established baseline. Use it when you have historical eval score distributions and need to determine whether a recent drop is statistically significant or random noise. The prompt ingests current eval results alongside baseline statistics and produces a structured regression alert with severity classification, affected dimensions, and prompt version correlation. It belongs in a continuous monitoring pipeline where eval scores flow in from automated evaluation runs and must be compared against known-good distributions before triggering on-call alerts.

The ideal deployment context assumes you already have eval infrastructure producing scored outputs and that baseline distributions are computed from a stable reference period—typically a rolling window of at least 100 evaluation runs with consistent prompt versions and model configurations. The prompt expects structured inputs including current eval scores, baseline mean and standard deviation per dimension, the number of samples in the current window, and the prompt version identifier. It performs statistical significance testing using z-score or t-test logic depending on sample size, then classifies severity into INFO, WARNING, or CRITICAL tiers based on the magnitude of deviation and the number of affected dimensions. The output includes a correlation check against recent prompt version changes to help isolate whether the regression coincides with a deployment event.

Do not use this prompt for real-time latency or error-rate monitoring; those require separate threshold-based alert templates such as the Latency Threshold Breach Detection or Tool-Call Error Rate Monitoring prompts. This prompt is also inappropriate when you lack a stable baseline—deploying it against a new eval dimension without sufficient historical data will produce unreliable severity classifications. If your eval scores are binary pass/fail rather than continuous distributions, use a proportion-based alert template instead. Finally, this prompt is not a replacement for root-cause investigation; it signals that a regression may exist, but diagnosing whether the cause is a model update, prompt change, retrieval degradation, or data drift requires follow-up trace analysis using the Eval Failure and Regression Investigation prompts.

The following prompt template is designed to be wired into a continuous monitoring pipeline. It receives a batch of recent eval scores, a historical baseline distribution, and metadata about active prompt versions. The model's job is not to decide whether to page someone—that decision belongs to your alerting infrastructure—but to produce a structured, evidence-rich analysis that your alerting system can threshold on and your on-call engineer can act on immediately. Replace every square-bracket placeholder with data from your observability store before sending the prompt to the model.

textCopy
You are an AI quality engineer analyzing production eval score data for regression detection.

## INPUT
Recent eval scores (last [EVALUATION_WINDOW] minutes):
[RECENT_EVAL_SCORES]

Historical baseline distribution (last [BASELINE_WINDOW] days):
[HISTORICAL_BASELINE]

Active prompt versions during the evaluation window:
[PROMPT_VERSION_MAP]

Eval dimensions being monitored:
[MONITORED_DIMENSIONS]

## TASK
Determine whether a statistically significant regression has occurred in any eval dimension by comparing the recent scores against the historical baseline.

## CONSTRAINTS
- Use a minimum significance threshold of [SIGNIFICANCE_THRESHOLD] (e.g., p < 0.05).
- Treat a drop below [PRACTICAL_SIGNIFICANCE_THRESHOLD] in mean score as practically significant even if statistical significance is borderline.
- If multiple dimensions show regression, rank them by severity.
- Correlate any regression onset with prompt version changes from [PROMPT_VERSION_MAP].
- Do not fabricate data. If the evidence is insufficient to conclude a regression, state that clearly.
- Flag any dimension where the sample size in [RECENT_EVAL_SCORES] is too small for reliable comparison.

## OUTPUT_SCHEMA
Return a single JSON object with this exact structure:
{
  "alert_triggered": boolean,
  "alert_severity": "critical" | "warning" | "info" | "none",
  "regression_summary": "string explaining the overall finding in one paragraph",
  "dimensions_affected": [
    {
      "dimension_name": "string",
      "baseline_mean": number,
      "recent_mean": number,
      "mean_drop": number,
      "p_value": number,
      "practical_significance_met": boolean,
      "sample_size_recent": number,
      "sample_size_baseline": number,
      "sample_size_adequate": boolean,
      "likely_cause": "string describing correlation with prompt version change or noting no clear cause"
    }
  ],
  "prompt_version_correlation": {
    "version_change_detected": boolean,
    "details": "string describing which versions changed and when, relative to regression onset"
  },
  "recommended_actions": ["string action items for the on-call engineer"],
  "caveats": ["string limitations or uncertainties in the analysis"]
}

## EXAMPLES
[FEW_SHOT_EXAMPLES]

Adapt this template by adjusting the significance thresholds to match your team's error budget and risk tolerance. If your eval dimensions are categorical rather than continuous, replace the mean-drop logic with proportion-shift logic. The [FEW_SHOT_EXAMPLES] placeholder should contain two or three example input-output pairs showing both a regression case and a no-regression case, which dramatically improves the model's calibration. Always validate the output JSON against the schema before forwarding the alert to your incident management system. If the model returns "sample_size_adequate": false for a critical dimension, suppress the alert and instead route the finding to a data quality review queue.

This prompt is designed to run as a scheduled evaluation, not as a real-time user-facing call. It consumes historical eval score distributions and recent production eval results, then produces a structured regression alert with statistical significance testing, affected dimensions, and prompt version correlation. The harness must supply the baseline distribution, the current window of scores, and metadata about which prompt versions were active during each period. Without accurate baseline data, the alert will either miss regressions or generate noise.

Wire the prompt into a monitoring pipeline that runs on a configurable cadence (e.g., hourly, after every N evaluations, or on each prompt version deployment). The harness should: (1) query your eval store for the baseline distribution—mean, variance, and sample count per eval dimension over a stable reference period; (2) query the current window of scores with timestamps and associated prompt version IDs; (3) populate the [BASELINE_DISTRIBUTION], [CURRENT_WINDOW], and [PROMPT_VERSION_MAP] placeholders; (4) call the model with temperature set to 0 and a strict JSON output schema; (5) validate the returned JSON against the expected alert schema before acting on it. If validation fails, retry once with the validation errors injected into [CONSTRAINTS]. If the second attempt fails, log the raw output and escalate to the on-call channel rather than silently dropping the alert.

For model choice, prefer a model with strong structured output and statistical reasoning capabilities. The prompt requires the model to interpret distributions, apply significance thresholds, and correlate regressions with prompt version changes—tasks where smaller or older models may hallucinate p-values or misattribute causes. Set response_format to json_schema (OpenAI) or equivalent structured output mode on your provider. Log every invocation with the input distributions, the raw model response, the validated alert payload, and whether the alert was suppressed, routed to a human, or published to an incident channel. This audit trail is essential when tuning alert thresholds or defending alert quality to stakeholders.

The alert output should feed into your incident management system. Map the severity field to your incident severity levels. Use the affected_dimensions and prompt_version_correlation fields to route the alert to the correct team (e.g., prompt engineering, retrieval, or model infrastructure). If the prompt returns requires_human_review: true, suppress automated paging and route to a review queue instead. Avoid wiring this prompt directly to auto-rollback systems—regression alerts can be caused by benign distribution shifts, and automated rollback without human review risks compounding the problem. Start with human-in-the-loop alerting, then automate responses only after you have calibrated false-positive rates over at least two full release cycles.