Score Drift Detection Prompt for Evaluation Pipelines

This prompt is built for MLOps and evaluation platform teams who need to catch silent scoring drift before it corrupts downstream decisions. The core job is comparing a new batch of judge scores against a stable calibration window and producing a structured drift report with severity flags, affected dimensions, and root cause hypotheses. You should use this when you have a well-defined baseline—typically a set of scores collected during a period of known-good judge behavior—and a new batch you suspect may be drifting due to model updates, prompt changes, or shifting input distributions. The prompt expects both the baseline and current score distributions, along with your configured drift thresholds and the evaluation dimensions you care about.

Do not use this prompt for initial judge calibration against human ratings. That workflow requires a different approach: mapping raw judge outputs to human annotator distributions and producing per-dimension alignment metrics. Similarly, do not use this for comparing two different judges against each other—inter-rater reliability prompts handle agreement coefficients, pairwise similarity, and outlier judge detection. This prompt assumes you already have a single judge you trust (or trusted) and need to know whether its behavior changed. It also assumes you are not doing real-time per-example scoring but rather batch-level statistical comparison. If you need per-example confidence intervals or uncertainty-aware scoring, use a judge confidence elicitation prompt instead.

Before running this prompt, ensure you have: a calibration window of scores with timestamps, a current batch of scores from the same judge and rubric, your organization's drift severity thresholds (e.g., what constitutes a warning vs. critical drift), and the list of evaluation dimensions you want checked independently. The prompt will return a structured report you can feed directly into monitoring dashboards or alerting systems. If the report flags critical drift, your next step is typically a root cause investigation—check whether the judge model was updated, the system prompt changed, or the input distribution shifted. Follow up with a judge instruction rewriting prompt if misalignment is confirmed, or a calibration feedback loop prompt if you need to rebuild alignment.

This prompt template is designed to be dropped into your evaluation monitoring pipeline whenever you need to check whether an LLM judge's scoring behavior has shifted from its calibrated baseline. It expects two structured inputs: a baseline calibration window containing historical scoring records with known-good human-verified labels, and a current evaluation window with recent judge outputs you want to audit. The prompt produces a drift severity flag, a per-dimension breakdown of where scores shifted, statistical significance indicators, and root cause hypotheses that help you decide whether to recalibrate, rewrite judge instructions, or investigate upstream data changes.

textCopy
You are an evaluation pipeline auditor specialized in detecting LLM judge score drift. Your task is to compare the current scoring behavior of an LLM judge against a historical baseline calibration window and produce a structured drift analysis.

## INPUTS

### Baseline Calibration Window
[BASELINE_RECORDS]

### Current Evaluation Window
[CURRENT_RECORDS]

### Judge Configuration
- Judge identifier: [JUDGE_ID]
- Evaluation dimensions monitored: [DIMENSIONS]
- Scoring scale: [SCORE_RANGE]
- Calibration window date range: [BASELINE_DATE_RANGE]
- Current window date range: [CURRENT_DATE_RANGE]

## DRIFT DETECTION PARAMETERS
- Minimum sample size per dimension for significance: [MIN_SAMPLE_SIZE]
- Drift severity thresholds:
  - Low drift: mean score shift < [LOW_THRESHOLD] standard deviations
  - Medium drift: mean score shift between [LOW_THRESHOLD] and [HIGH_THRESHOLD] standard deviations
  - High drift: mean score shift > [HIGH_THRESHOLD] standard deviations
- Statistical test: [TEST_METHOD]
- Significance level (alpha): [ALPHA]

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "drift_summary": {
    "overall_drift_detected": boolean,
    "overall_severity": "none" | "low" | "medium" | "high",
    "dimensions_with_drift": [string],
    "comparison_periods": {
      "baseline_window": string,
      "current_window": string,
      "baseline_sample_count": number,
      "current_sample_count": number
    }
  },
  "per_dimension_analysis": [
    {
      "dimension": string,
      "baseline_mean": number,
      "current_mean": number,
      "mean_shift": number,
      "shift_in_std_devs": number,
      "baseline_std": number,
      "current_std": number,
      "p_value": number,
      "statistically_significant": boolean,
      "severity": "none" | "low" | "medium" | "high",
      "sample_size_baseline": number,
      "sample_size_current": number,
      "distribution_shape_change": string,
      "score_bucket_shifts": [
        {
          "bucket": string,
          "baseline_proportion": number,
          "current_proportion": number,
          "delta": number
        }
      ]
    }
  ],
  "root_cause_hypotheses": [
    {
      "hypothesis": string,
      "confidence": "low" | "medium" | "high",
      "supporting_evidence": string,
      "affected_dimensions": [string],
      "recommended_investigation": string
    }
  ],
  "recommended_actions": [
    {
      "action": string,
      "priority": "immediate" | "next_cycle" | "monitor",
      "rationale": string,
      "trigger_condition": string
    }
  ],
  "calibration_health": {
    "recalibration_recommended": boolean,
    "urgency": "none" | "routine" | "elevated" | "critical",
    "minimum_samples_needed_for_recalibration": number,
    "dimensions_needing_new_anchor_examples": [string]
  }
}

## CONSTRAINTS
- Do not flag drift when sample size is below [MIN_SAMPLE_SIZE] per dimension. Mark those dimensions as "insufficient data" and exclude from severity calculations.
- When p_value exceeds [ALPHA], set statistically_significant to false and severity to "none" regardless of mean shift magnitude.
- For root cause hypotheses, only propose causes that can be distinguished by the available data. Do not speculate about model updates unless version metadata is provided.
- If the baseline window contains fewer than [MIN_SAMPLE_SIZE] records total, return an error object: {"error": "insufficient_baseline_data", "minimum_required": [MIN_SAMPLE_SIZE], "provided": count}.
- Preserve all numeric precision to 4 decimal places.
- If score distributions show compression toward the mean in the current window, flag this as a potential "regression to safe scores" pattern in root cause hypotheses.

## ANALYSIS INSTRUCTIONS
1. Group records by evaluation dimension.
2. For each dimension with sufficient samples, compute baseline and current mean, standard deviation, and score bucket proportions.
3. Apply [TEST_METHOD] to test for significant differences.
4. Classify severity per dimension using the configured thresholds.
5. Generate root cause hypotheses by examining patterns across dimensions: simultaneous drift in correlated dimensions, drift direction consistency, and relationship to input characteristics available in the records.
6. Prioritize recommended actions based on severity and downstream decision impact.

To adapt this template for your pipeline, replace the square-bracket placeholders with your actual configuration values. The [BASELINE_RECORDS] and [CURRENT_RECORDS] fields should contain serialized JSON arrays of scoring records, each including at minimum the dimension name, score value, and any available metadata about the input that was scored. If your judge operates on a 1-5 Likert scale, set [SCORE_RANGE] to "1-5" and adjust [LOW_THRESHOLD] and [HIGH_THRESHOLD] accordingly—typical starting values are 0.5 and 1.5 standard deviations. For [TEST_METHOD], use "Mann-Whitney U" when score distributions are non-normal or ordinal, or "Welch's t-test" when distributions are approximately normal and sample sizes differ. Always set [MIN_SAMPLE_SIZE] to at least 30 per dimension per window to avoid false drift alarms from small-sample noise. Before deploying, validate that your serialized record format matches what the prompt expects by running a single dry-run with known-stable data and confirming the output parses correctly into your monitoring dashboard.

This prompt is designed to run as a scheduled batch job, not a real-time endpoint. Wire it into your evaluation pipeline by triggering it after every N evaluation runs or on a fixed cadence (daily, weekly). The primary inputs are two structured datasets: a baseline calibration window (scores from a known-stable period, typically 200-500 scored examples with human-verified labels) and a current observation window (the most recent batch of LLM judge scores, same dimensions). Both must include per-dimension scores, confidence values if available, and metadata about the input difficulty tier. Store these in your evaluation database and pass them as serialized JSON arrays into the [BASELINE_SCORES] and [CURRENT_SCORES] placeholders.

Before calling the model, run statistical pre-checks in your application code: compute basic distribution statistics (mean, variance, percentiles) for each scoring dimension in both windows. If the current window has fewer than [MIN_SAMPLE_SIZE] examples, skip the LLM call and log an insufficient-data warning. If a Kolmogorov-Smirnov or Mann-Whitney U test already shows no significant difference at p < [SIGNIFICANCE_THRESHOLD], you can optionally skip the LLM analysis to save cost, but the prompt's value is in producing root cause hypotheses that statistical tests alone won't surface. Pass the pre-computed statistics into [PRE_COMPUTED_STATS] to ground the model's reasoning in actual numbers rather than asking it to hallucinate distribution properties.

Output validation is critical because this prompt feeds into alerting systems. Parse the JSON response and validate: (1) drift_severity is one of the allowed enum values, (2) affected_dimensions contains only dimension names present in your input schema, (3) statistical_significance fields are numeric between 0 and 1, and (4) root_cause_hypotheses contains at least one entry if drift severity is not 'none'. If validation fails, retry once with the validation errors appended to the [CONSTRAINTS] field. If the retry also fails, log the raw output and escalate to a human reviewer rather than silently accepting a malformed drift report.

Model choice matters. Use a model with strong quantitative reasoning (GPT-4o, Claude 3.5 Sonnet, or equivalent) because the prompt requires comparing distributions, interpreting statistical signals, and avoiding false alarms. Set temperature to 0 or near-zero for reproducibility. If your evaluation pipeline processes high volumes, consider routing low-risk score ranges (where drift is unlikely to matter) to a faster, cheaper model and reserving the full analysis for dimensions where scores are near decision boundaries. Log every drift detection result with the prompt version, model identifier, input window sizes, and validation status for auditability.

Alerting integration should use the structured output fields directly: trigger a warning alert if drift_severity is 'low', a critical alert if 'medium' or 'high', and an automated rollback or judge retraining ticket if 'critical'. Include the affected_dimensions and top root_cause_hypotheses in the alert payload so on-call engineers don't have to open the raw report. Avoid alerting on drift in dimensions that are not used in downstream decisions—filter by [MONITORED_DIMENSIONS] before sending alerts. Finally, schedule a periodic review of drift detection results with your evaluation lead to identify whether recurring drift patterns indicate a need for judge instruction rewrites or calibration set updates.