Confidence Drift Monitoring Prompt

This prompt is designed for MLOps engineers and AI platform teams who need to automate the detection of distributional shifts in a production classifier's confidence scores. The core job-to-be-done is to compare a recent batch of classification outputs against a known-good historical baseline and produce a structured, actionable drift report. This is not a prompt for real-time, single-input classification. It is a scheduled batch monitoring tool intended to run on a fixed interval—such as hourly or daily—against aggregated classification logs. The ideal user already collects confidence scores per classification event and can supply both a current distribution summary and a reference baseline distribution summary as structured inputs.

Use this prompt when you have a stable classification system in production and you need an automated check for silent degradation. For example, if a model's accuracy remains acceptable but its confidence calibration has drifted—producing overconfident low-quality predictions or underconfident high-quality ones—this prompt can flag that shift before it causes downstream routing errors. It is appropriate when you can define a statistical comparison method (e.g., KL divergence, KS test, Wasserstein distance) and a threshold for what constitutes a meaningful drift. You must provide the current window's confidence score distribution and the baseline distribution in a structured format, such as binned histograms or summary statistics (mean, variance, percentiles). The prompt assumes you have already segmented your data by class label if per-class drift analysis is required.

Do not use this prompt for real-time inference gating, as it is not designed to make per-event accept/reject decisions. It is also not a replacement for a full model monitoring platform; it is a focused analytical step that can feed into your existing alerting and observability stack. Avoid using it when the baseline distribution is unstable or poorly defined, as this will generate noisy, unactionable alerts. The next step after reading this section is to prepare your distribution inputs and define your drift severity thresholds before wiring the prompt into your monitoring harness.

This prompt is designed to be executed on a scheduled basis within your MLOps pipeline. It takes a snapshot of recent classification confidence scores and a reference baseline, then produces a structured drift assessment. The output is intended for automated alerting systems, not just human review. Replace every square-bracket placeholder with data from your monitoring store before execution.

textCopy
You are a production ML monitoring agent. Your task is to compare a current window of classification confidence scores against a historical baseline distribution and determine if statistically significant drift has occurred.

## INPUT DATA
- Current window scores: [CURRENT_SCORES]
- Baseline scores: [BASELINE_SCORES]
- Classification labels associated with scores: [LABELS]
- Timestamp range for current window: [CURRENT_WINDOW_START] to [CURRENT_WINDOW_END]
- Timestamp range for baseline: [BASELINE_START] to [BASELINE_END]

## DRIFT DETECTION CRITERIA
Analyze the distributions using these signals:
1. Mean shift: Has the average confidence changed by more than [MEAN_SHIFT_THRESHOLD]?
2. Variance change: Has the spread of scores widened or narrowed by more than [VARIANCE_THRESHOLD]?
3. Low-confidence spike: Has the proportion of scores below [LOW_CONFIDENCE_THRESHOLD] increased by more than [LOW_CONF_PROPORTION_THRESHOLD]?
4. High-confidence collapse: Has the proportion of scores above [HIGH_CONFIDENCE_THRESHOLD] dropped by more than [HIGH_CONF_PROPORTION_THRESHOLD]?
5. Per-label drift: For each label in [LABELS], check if its score distribution has shifted independently.

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "drift_detected": boolean,
  "overall_severity": "none" | "low" | "medium" | "high" | "critical",
  "drift_signals": [
    {
      "signal_type": "mean_shift" | "variance_change" | "low_confidence_spike" | "high_confidence_collapse" | "per_label_drift",
      "detected": boolean,
      "current_value": number,
      "baseline_value": number,
      "threshold": number,
      "label": string | null,
      "description": string
    }
  ],
  "affected_labels": [string],
  "recommended_action": "none" | "log_and_continue" | "increase_sampling" | "trigger_retraining_review" | "rollback_model" | "escalate_to_oncall",
  "action_rationale": string,
  "summary": string
}

## CONSTRAINTS
- Do not hallucinate statistical tests you cannot compute from the provided score arrays.
- If the current window has fewer than [MIN_SAMPLE_SIZE] scores, set drift_detected to false and severity to "none" with a note in the summary.
- Compare distributions using the thresholds provided; do not invent new thresholds.
- If per-label sample sizes are too small for reliable comparison, flag those labels in affected_labels with a note in the description.
- The recommended_action must be one of the enumerated values only.

Adaptation guidance: Wire this prompt into a scheduled job that queries your monitoring database for the current window and baseline scores. The thresholds should be tuned per model using historical false-positive rates. For high-risk classification systems, add a human review step before automated rollback actions. Validate the output JSON against the schema before acting on recommended_action. Log every drift assessment with the full prompt, response, and timestamp for auditability.

The Confidence Drift Monitoring Prompt is designed to be called on a schedule—hourly, daily, or per batch window—not in the hot path of user requests. The prompt compares a current window of classification outputs (label, confidence score, input summary) against a historical baseline distribution. The harness must therefore manage two data sources: a baseline reference (a frozen snapshot of expected confidence distributions per intent) and a current observation window (the most recent N inferences). The prompt itself performs the comparison and generates a drift report, but the surrounding application code is responsible for data preparation, validation, and acting on the results.

Data preparation and validation are the first integration points. Before calling the model, the harness must: (1) aggregate raw inference logs into the [CURRENT_WINDOW] format—typically a JSON array of objects with intent, confidence, and input_id fields; (2) load the [BASELINE_DISTRIBUTION] from a trusted store (a database, a versioned artifact, or a config file); (3) validate that both datasets contain the same intent taxonomy and that the current window meets minimum sample size thresholds (e.g., at least 50 inferences per intent before running drift detection). If the current window is too small, the harness should skip the prompt call and log an insufficient_data event rather than producing a noisy or misleading drift alert. The prompt template expects these two structured inputs plus a [DRIFT_THRESHOLD] parameter (e.g., a Wasserstein distance or KS statistic threshold) that defines what constitutes actionable drift.

Post-prompt validation and alerting close the loop. The prompt's output schema should include a drift_detected boolean, a drift_severity enum (none, low, medium, high), per-intent drift scores, and a recommended_action field. The harness must validate this output against the expected schema before acting on it. If the model returns malformed JSON or missing required fields, the harness should retry once with a repair prompt (see Output Repair and Validation Prompts) and then escalate to a human operator if the retry fails. When drift is detected at medium or high severity, the harness should: (a) write the full drift report to an observability store (e.g., a dedicated database table or log index); (b) fire an alert to the MLOps team's incident channel with the drift severity, affected intents, and a link to the report; and (c) optionally trigger an automated rollback or model refresh pipeline if the drift exceeds a critical threshold. For low severity drift, log the report without paging anyone—this builds a history for trend analysis without causing alert fatigue.

Eval checks and calibration are essential before trusting this prompt in production. Build a regression test suite with synthetic drift scenarios: inject known distribution shifts (e.g., drop the mean confidence of a specific intent by 0.2, or introduce a bimodal confidence pattern) and verify that the prompt detects the drift at the expected severity level. Also test negative cases: feed the prompt two statistically identical distributions and confirm it reports drift_detected: false. Measure the prompt's sensitivity to sample size—does it over-flag drift when the current window is small? If so, enforce a minimum sample size in the harness. Finally, periodically backtest the prompt against historical production data where known drift events occurred (e.g., after a model update or a data pipeline change) to ensure the detection logic remains calibrated to your actual operating conditions. The prompt is a signal generator; the harness is responsible for making that signal reliable, actionable, and non-disruptive to on-call teams.