Calibration Drift Monitoring Prompt Over Time Windows

This prompt is for evaluation ops teams running continuous judge monitoring pipelines who need to answer a specific question: is our judge getting worse over time, how fast, and when will it break our decision thresholds? It compares calibration quality across two or more sequential time windows—a baseline calibration window and one or more current evaluation windows—and produces drift trends, acceleration warnings, and projected degradation timelines. The prompt assumes you already have per-dimension alignment metrics (such as mean absolute error, correlation coefficients, or agreement rates against human labels) for each window. It does not replace initial judge calibration, single-window drift detection, or root cause analysis of individual score discrepancies.

Use this prompt when you have at least two completed evaluation windows with matching metric structures, and you need to surface trends rather than point-in-time drift flags. The ideal user is an evaluation infrastructure engineer or ML ops lead who already runs periodic calibration checks and now wants to automate the longitudinal analysis. Required inputs include: a baseline window identifier with its per-dimension metrics, one or more comparison windows with the same metric schema, the decision thresholds that matter for your product (e.g., pass/fail boundaries, score tolerance bands), and the business impact of threshold violations. The prompt produces a structured drift trend report with acceleration detection, projected threshold-crossing dates, and recommended alert levels. Do not use this prompt for initial judge calibration, single-window drift detection, root cause analysis of individual score discrepancies, or real-time scoring decisions. Those workflows have their own dedicated prompts in this content group.

Before running this prompt, validate that your input windows are comparable: they must use the same judge configuration, the same evaluation dimensions, and the same metric types. If you changed judge instructions, scoring rubrics, or evaluation datasets between windows, the trend analysis will be misleading. The prompt includes explicit checks for window comparability and will flag structural mismatches before producing trend conclusions. For high-stakes decision thresholds—such as automated pass/fail gates in CI/CD pipelines or content moderation systems—the output includes a recommendation for human review of any projected threshold crossing that falls within your specified risk horizon. After receiving the drift trend report, your next step is typically to trigger a root cause analysis prompt if acceleration is detected, or to update your judge calibration if thresholds are projected to fail within an unacceptable timeframe.

This prompt template is designed to be pasted directly into your evaluation monitoring workflow. It instructs an LLM to act as a calibration auditor, comparing judge scoring behavior between a baseline window and a recent window. The prompt expects structured input containing scored examples from both windows, along with human reference labels or a trusted gold standard. Replace every square-bracket placeholder with your actual data before execution. The output is a structured drift report, not a conversational summary, making it suitable for automated ingestion into dashboards, alerting systems, or CI/CD gates.

codeCopy
You are an evaluation calibration auditor. Your task is to compare the calibration quality of an LLM judge between two sequential time windows and produce a structured drift report.

## INPUT DATA

### Baseline Window
- Time range: [BASELINE_START_DATE] to [BASELINE_END_DATE]
- Number of scored examples: [BASELINE_SAMPLE_COUNT]
- Scored examples with human reference labels:
[BASELINE_EXAMPLES]
  Format per example: {"id": "[EXAMPLE_ID]", "input": "[INPUT_TEXT]", "output": "[MODEL_OUTPUT]", "judge_score": [JUDGE_SCORE], "human_score": [HUMAN_SCORE], "dimension": "[EVALUATION_DIMENSION]"}

### Recent Window
- Time range: [RECENT_START_DATE] to [RECENT_END_DATE]
- Number of scored examples: [RECENT_SAMPLE_COUNT]
- Scored examples with human reference labels:
[RECENT_EXAMPLES]
  Format per example: same as baseline

## OUTPUT SCHEMA

Return a valid JSON object with this exact structure:
{
  "drift_detected": boolean,
  "drift_severity": "none" | "low" | "medium" | "high" | "critical",
  "overall_calibration_shift": {
    "baseline_correlation": number,
    "recent_correlation": number,
    "correlation_delta": number,
    "mean_score_delta": number,
    "interpretation": "string"
  },
  "per_dimension_drift": [
    {
      "dimension": "string",
      "baseline_correlation": number,
      "recent_correlation": number,
      "drift_detected": boolean,
      "drift_magnitude": number,
      "direction": "over_scoring" | "under_scoring" | "no_shift",
      "sample_count_baseline": number,
      "sample_count_recent": number
    }
  ],
  "acceleration_warning": boolean,
  "acceleration_details": "string or null",
  "projected_degradation": {
    "estimated_days_until_critical": number or null,
    "confidence": "low" | "medium" | "high",
    "basis": "string"
  },
  "alert_threshold_recommendations": {
    "low_severity_threshold": number,
    "medium_severity_threshold": number,
    "high_severity_threshold": number,
    "rationale": "string"
  },
  "recommended_actions": ["string"],
  "statistical_significance": {
    "test_used": "string",
    "p_value": number or null,
    "significant_at_0_05": boolean,
    "caveats": "string"
  }
}

## CONSTRAINTS

- Compare calibration quality using correlation between judge scores and human reference scores within each window.
- Detect drift by measuring whether the judge's scoring behavior changed between windows relative to human labels.
- Flag acceleration when the rate of drift in the recent window exceeds the historical rate.
- Recommend alert thresholds based on the downstream decision impact level: [DECISION_IMPACT_LEVEL]. Use "low" for informational dashboards, "medium" for release gates, "high" for safety-critical or regulated decisions.
- If sample counts are below [MINIMUM_SAMPLE_THRESHOLD] per window, set confidence to "low" and note the limitation in caveats.
- Do not hallucinate correlation values. If data is insufficient for a dimension, set values to null and note the gap.
- Treat human scores as ground truth. Do not second-guess human labels.

## EVALUATION DIMENSIONS
[EVALUATION_DIMENSIONS_LIST]

## RISK LEVEL
[RISK_LEVEL]

After copying the template, adapt it by replacing the placeholders with your actual calibration data. The [BASELINE_EXAMPLES] and [RECENT_EXAMPLES] arrays should contain scored examples with both judge and human labels for the same set of evaluation dimensions. If you lack human labels for the recent window, this prompt is not appropriate—use the Score Drift Detection Prompt instead. Set [DECISION_IMPACT_LEVEL] honestly: over-alerting on low-impact dashboards wastes engineering time, while under-alerting on safety-critical gates can cause real harm. The [MINIMUM_SAMPLE_THRESHOLD] should reflect your statistical power requirements; fewer than 30 examples per dimension per window typically produces unreliable drift signals. Before deploying this prompt into an automated pipeline, run it against a known-stable period to verify it does not produce false-positive drift alerts, and validate the output JSON against the schema to catch parsing failures early.

This prompt is designed to run as a scheduled evaluation job, not a one-off analysis. Wire it into your existing evaluation pipeline so it executes at a fixed cadence—daily, weekly, or per evaluation batch—depending on how fast your judge fleet produces scores. The prompt expects two structured inputs: a baseline calibration window and a current evaluation window, each containing scored examples with human ratings and LLM judge outputs. These inputs should be assembled programmatically from your evaluation store before invoking the model. The output is a structured drift report with severity flags, affected dimensions, and projected degradation timelines, which should be parsed and routed to your monitoring dashboard or alerting system.

Build the harness with these concrete checks. Input validation: before calling the model, verify that both windows contain minimum sample sizes per dimension (at least 30 scored examples per dimension to avoid noisy drift signals), that score distributions aren't degenerate (all 4s or all 1s), and that timestamps are monotonically increasing. Reject malformed windows with a clear error before spending inference tokens. Model selection: use a model with strong structured output capabilities and a context window large enough to hold both windows plus the analysis instructions—Claude 3.5 Sonnet or GPT-4o are appropriate for production; avoid smaller models that may hallucinate statistical claims. Set temperature=0 for deterministic outputs and use structured output mode (JSON mode or tool calling) to enforce the expected schema. Retry logic: if the model returns malformed JSON or missing required fields, retry once with the error message appended. If the second attempt fails, log the raw output and escalate to a human reviewer rather than silently dropping the analysis. Logging: capture the prompt version, model ID, window date ranges, sample counts per dimension, raw model response, parsed drift report, and any validation errors. This audit trail is essential when stakeholders question a drift alert.

After parsing the model's output, implement threshold-based alerting in your application layer, not in the prompt. The prompt produces drift severity flags and recommended thresholds, but your harness should enforce the actual alerting rules. For example: if drift_severity is high on any dimension, trigger a P2 alert; if acceleration_warning is true, escalate to P1. Route alerts to the evaluation platform on-call channel with a link to the raw analysis. Avoid over-alerting by requiring at least two consecutive windows with elevated drift before paging—single-window noise is common. Finally, store each drift report in your evaluation database with the window ranges as the primary key, enabling trend queries and historical comparison. Do not rely on the model's projected degradation timeline as a scheduling trigger; use it as a directional signal while your own monitoring cadence remains the source of truth for when the next check runs.