Judge Drift Monitoring Over Time Prompt Template

Use this prompt when you operate a fleet of LLM judges and need to detect meaningful shifts in their scoring behavior over time. The primary job is to transform raw time-series score data into actionable drift alerts, not to evaluate individual outputs. This prompt is designed for evaluation platform teams, MLOps engineers, and AI infrastructure operators who already have a running judge pipeline and need statistical process control over its output quality. The ideal user has access to historical judge scores, understands their evaluation rubric, and can configure alert thresholds based on their application's tolerance for variance.

This prompt is not a replacement for human review of individual evaluations, nor is it suitable for one-off score comparisons. Do not use it when you lack sufficient historical data to establish a baseline distribution—at minimum, you need several time windows of scores before drift detection becomes statistically meaningful. Avoid this prompt if your judges are still in active calibration or if you are testing a new rubric, because early-stage volatility will generate false alarms that erode trust in the monitoring system. The prompt assumes your input data is already clean, timestamped, and grouped by judge identifier; it does not perform data cleaning or missing-value imputation.

Before wiring this into production, ensure you have defined your alert severity levels, false-alarm suppression windows, and escalation paths. The prompt produces a structured drift report, but your application harness must handle alert routing, deduplication, and human review workflows. If your evaluation pipeline is high-stakes—such as safety scoring, regulatory compliance checks, or customer-facing quality gates—always pair automated drift alerts with a human-in-the-loop review step before taking corrective action on the judge fleet.

This prompt template is designed to be dropped into an evaluation monitoring pipeline that feeds historical judge score distributions into an LLM for drift analysis. It expects time-series score data, configurable alert thresholds, and a defined output schema so that downstream automation can act on drift alerts without manual interpretation. The template uses square-bracket placeholders for all variable inputs, making it safe to copy, adapt, and parameterize in code before each monitoring run.

textCopy
You are a statistical process control analyst monitoring an LLM judge fleet for scoring drift over time. Your task is to compare score distributions across two time windows, detect meaningful drift, classify its severity, and recommend actions.

## INPUT DATA

### Current Window
- Time range: [CURRENT_WINDOW_START] to [CURRENT_WINDOW_END]
- Number of evaluations: [CURRENT_SAMPLE_SIZE]
- Score distribution: [CURRENT_SCORE_DISTRIBUTION]
- Mean score: [CURRENT_MEAN]
- Standard deviation: [CURRENT_STD]
- Judge identifier: [JUDGE_ID]
- Evaluation criteria: [CRITERIA_LIST]

### Baseline Window
- Time range: [BASELINE_WINDOW_START] to [BASELINE_WINDOW_END]
- Number of evaluations: [BASELINE_SAMPLE_SIZE]
- Score distribution: [BASELINE_SCORE_DISTRIBUTION]
- Mean score: [BASELINE_MEAN]
- Standard deviation: [BASELINE_STD]

## DRIFT DETECTION PARAMETERS
- Alert threshold (z-score): [ALERT_Z_THRESHOLD]
- Warning threshold (z-score): [WARNING_Z_THRESHOLD]
- Minimum sample size for analysis: [MIN_SAMPLE_SIZE]
- Suppression window (hours): [SUPPRESSION_WINDOW_HOURS]
- False alarm sensitivity: [FALSE_ALARM_SENSITIVITY]

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "drift_detected": boolean,
  "severity": "none" | "warning" | "alert" | "critical",
  "drift_metrics": {
    "mean_shift": number,
    "mean_shift_z_score": number,
    "distribution_divergence": number,
    "divergence_method": "ks_statistic" | "wasserstein" | "chi_squared",
    "variance_change_ratio": number
  },
  "affected_criteria": [
    {
      "criterion": string,
      "drift_magnitude": number,
      "direction": "increase" | "decrease" | "no_change"
    }
  ],
  "statistical_significance": {
    "p_value": number,
    "confidence_interval_lower": number,
    "confidence_interval_upper": number,
    "sample_size_sufficient": boolean
  },
  "false_alarm_risk": "low" | "medium" | "high",
  "recommended_actions": [string],
  "suppression_recommendation": {
    "should_suppress": boolean,
    "suppress_until": string | null,
    "reason": string
  },
  "diagnostic_hypotheses": [
    {
      "hypothesis": string,
      "likelihood": "low" | "medium" | "high",
      "evidence": string
    }
  ]
}

## CONSTRAINTS
- Do not flag drift when sample sizes are below [MIN_SAMPLE_SIZE]. Set sample_size_sufficient to false and severity to "none".
- If the current window overlaps with a suppression window from a previous alert, set should_suppress to true and explain why.
- Distinguish between mean shift (systematic bias) and distribution shape change (increased variance or multimodality).
- When false_alarm_risk is "high", severity must not exceed "warning" regardless of z-score.
- For each affected criterion, cite the specific score change observed.
- Diagnostic hypotheses must be grounded in the observed data patterns, not speculation.
- If divergence_method cannot be reliably computed, use "ks_statistic" as default and note the limitation.

## EXAMPLES

### Example 1: Clear Drift
Current mean: 3.8, Baseline mean: 4.2, z-score: 2.8
Expected: drift_detected=true, severity="alert", direction="decrease"

### Example 2: No Drift
Current mean: 4.15, Baseline mean: 4.2, z-score: 0.3
Expected: drift_detected=false, severity="none"

### Example 3: Small Sample
Current sample: 12, Baseline sample: 200
Expected: sample_size_sufficient=false, drift_detected=false

### Example 4: Suppression Active
Current window falls within prior suppression window
Expected: should_suppress=true, severity capped at "warning"

To adapt this template, replace each square-bracket placeholder with values from your monitoring database or configuration store. The [CURRENT_SCORE_DISTRIBUTION] and [BASELINE_SCORE_DISTRIBUTION] fields should contain binned score counts or raw arrays depending on your statistical method preference. Wire the output JSON into your alerting system so that severity: "critical" triggers an on-call page while severity: "warning" creates a dashboard annotation. Before deploying, run this prompt against a golden dataset of known drift and no-drift windows to verify that your chosen thresholds produce acceptable false-positive and false-negative rates.

The Judge Drift Monitoring prompt is designed to run as a scheduled batch job, not a real-time call. You feed it a time-series dataset of judge scores across two windows (baseline and current) and receive a structured drift alert payload. The harness must handle data preparation, prompt invocation, output validation, and integration with your alerting or dashboarding systems. Because drift detection can generate false alarms from normal variance, the implementation must include configurable thresholds, suppression windows, and human-review gates before any automated action is taken.

Start by preparing the input data. Aggregate judge scores into a JSON array where each record includes judge_id, score, timestamp, and optional item_id and rubric_criterion. Split the data into two windows: a baseline window (e.g., the previous 7 days) and a current window (e.g., the last 24 hours). The prompt expects these as [BASELINE_SCORES] and [CURRENT_SCORES] arrays. Before calling the model, run a statistical pre-check: if the current window has fewer than 30 scores or the baseline window has fewer than 100, skip the LLM call and log an INSUFFICIENT_DATA status. This prevents the model from hallucinating drift signals from noise. For model choice, use a model with strong numerical reasoning and structured output support (GPT-4o or Claude 3.5 Sonnet work well). Set temperature=0 and enforce the output schema with your model's JSON mode or structured output feature. The prompt returns a JSON object with drift_detected, severity, affected_judges, before_after_comparison, and recommended_actions fields.

After receiving the response, validate the output against a strict schema. Check that severity is one of NONE, LOW, MEDIUM, HIGH, or CRITICAL. Verify that affected_judges is an array of objects with judge_id, drift_magnitude, and direction fields. If validation fails, retry once with the same prompt and a note about the schema violation. If it fails again, log the failure and escalate to a human reviewer. For alerting, implement a suppression rule: if the same judge triggers a MEDIUM or lower alert within a 48-hour window, suppress the duplicate alert and increment a suppression counter. Only fire a new alert when severity reaches HIGH or CRITICAL, or when a suppressed alert has fired more than 3 times in a week. This prevents alert fatigue. Log every prompt invocation, the raw response, validation results, and any suppression decisions to your observability platform for audit and debugging.

Finally, wire the drift alerts into your evaluation dashboard and incident management system. For HIGH and CRITICAL alerts, create a ticket or incident with the affected judge IDs, the before/after comparison, and a link to the raw score distributions. Include a runbook step that instructs the responder to check for known causes: model version changes, prompt template updates, or upstream data pipeline issues. Do not automatically disable judges based on drift alerts alone. Always require human confirmation before removing a judge from the evaluation fleet. This harness turns the prompt from a one-off analysis into a reliable, production-grade monitoring signal.