Intent Drift Detection Prompt for Production Monitoring

This playbook is for MLOps engineers and AI platform teams who already have an intent classifier running in production and need a systematic way to detect when the distribution of incoming requests shifts away from the baseline. The job-to-be-done is production monitoring: you sample recent traffic, compare it against a known reference distribution, and produce a structured drift assessment that tells you which intent categories have moved beyond acceptable thresholds. This is not a prompt for classifying individual user messages. It is a diagnostic tool that sits in your observability pipeline, running on a schedule or triggered by a deployment event, to answer the question: 'Has user behavior, input patterns, or classifier performance changed enough that we need to investigate?'

The ideal user has access to a labeled baseline distribution—typically derived from a golden dataset, a stable production window, or a curated evaluation set—and a current sample drawn from recent production logs. You will need to provide both distributions as structured input, along with your acceptable drift thresholds per category and any taxonomy constraints. The prompt assumes you are comparing categorical distributions, not raw embeddings or continuous scores. It works best when your intent taxonomy is stable and your baseline is representative. If your taxonomy has changed significantly between baseline and sample, this prompt will flag that as a structural break rather than a distributional shift, which is a different class of problem requiring taxonomy migration tooling rather than drift detection.

Do not use this prompt for real-time per-request classification or inline routing decisions. It is designed for batch analysis of sampled production data against a baseline, not for low-latency inference paths. It is also not a replacement for embedding-based drift detection, statistical process control charts, or population stability index (PSI) calculations run outside the model. Use this prompt when you need a human-readable, structured assessment that an on-call engineer or a downstream automation can act on—such as triggering a model retraining pipeline, updating routing weights, or escalating to a product manager for taxonomy review. If you need sub-second latency or continuous streaming drift detection, implement a statistical method in your application layer and use this prompt for periodic interpretive summaries.

The prompt below is designed to be wired into a production monitoring pipeline. It expects a statistical summary of current production classifications alongside a reference baseline, and it returns a structured drift assessment. Use it when you need a human-readable, actionable report from raw distribution data—not when you need the statistical computation itself, which should happen in your application code before calling the model.

textCopy
You are an MLOps drift analyst reviewing intent classifier performance in production. Your job is to compare the current production distribution of classified intents against a known baseline distribution, identify meaningful drift, and recommend whether retraining or investigation is needed.

## INPUTS
- Baseline Distribution: [BASELINE_DISTRIBUTION]
- Current Production Distribution: [CURRENT_DISTRIBUTION]
- Drift Metrics: [DRIFT_METRICS]
- Observation Window: [OBSERVATION_WINDOW]
- Alert Thresholds: [ALERT_THRESHOLDS]
- Intent Taxonomy: [INTENT_TAXONOMY]

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "drift_detected": boolean,
  "overall_drift_score": number,
  "assessment": string,
  "flagged_categories": [
    {
      "intent": string,
      "baseline_frequency": number,
      "current_frequency": number,
      "drift_magnitude": number,
      "direction": "increase" | "decrease",
      "likely_cause": string,
      "severity": "low" | "medium" | "high" | "critical"
    }
  ],
  "unseen_intents": [string],
  "recommendations": [
    {
      "action": "retrain" | "investigate" | "monitor" | "update_taxonomy" | "no_action",
      "rationale": string,
      "urgency": "immediate" | "next_cycle" | "routine"
    }
  ],
  "retraining_trigger": boolean,
  "retraining_rationale": string
}

## CONSTRAINTS
- Only flag categories where drift exceeds the provided alert thresholds.
- If an intent appears in the current distribution but not in the baseline, list it under `unseen_intents` and recommend a taxonomy update.
- If an intent from the baseline has dropped to near-zero frequency, flag it with direction "decrease" and investigate whether the classifier is failing or user behavior has genuinely shifted.
- Do not fabricate causes. Base `likely_cause` on distribution patterns, not speculation about external events.
- If overall drift is below thresholds, set `drift_detected` to false and `retraining_trigger` to false.
- Use the observation window to contextualize whether drift is sustained or transient.

## EXAMPLES
Example flagged category:
{
  "intent": "account_cancellation",
  "baseline_frequency": 0.12,
  "current_frequency": 0.04,
  "drift_magnitude": 0.08,
  "direction": "decrease",
  "likely_cause": "Possible classifier confusion with 'billing_dispute' intent, which shows a corresponding increase.",
  "severity": "high"
}

## RISK LEVEL
[RISK_LEVEL]

To adapt this prompt, replace each square-bracket placeholder with data from your monitoring pipeline. [BASELINE_DISTRIBUTION] and [CURRENT_DISTRIBUTION] should be pre-computed frequency tables keyed by intent label. [DRIFT_METRICS] should include the statistical method used—such as Jensen-Shannon divergence, PSI, or chi-squared—along with computed values. [ALERT_THRESHOLDS] defines the numeric boundaries that trigger flags. [RISK_LEVEL] should be set to "high" when misclassification carries regulatory, financial, or safety consequences; this signals the model to apply stricter criteria and recommend human review of flagged categories before automated retraining begins.

After copying the template, validate that your application layer computes all statistical inputs before calling the model. The model should reason about drift, not calculate it. Test the prompt with known drift scenarios—including no-drift baselines, single-category shifts, and sudden appearance of unseen intents—and confirm that retraining_trigger fires only when justified. Log every assessment with the input distributions and model output for auditability. If the prompt is used in a regulated domain, route critical severity flags to a human reviewer before triggering automated retraining pipelines.

The Intent Drift Detection prompt is not a one-shot query; it is a scheduled evaluation step inside an MLOps pipeline. The harness must fetch a representative sample of recent production inputs, run them through the same intent classifier being monitored, and pass the resulting distribution alongside a stored baseline distribution to the drift detection prompt. The prompt returns a structured drift assessment, not a raw narrative. The harness should parse this assessment into a machine-readable format (JSON) and feed it into your observability stack—Grafana, Datadog, or a custom dashboard—as a drift event with severity, flagged categories, and recommended actions.

A concrete implementation loop runs on a cadence (daily or weekly) and follows these steps: (1) Sample extraction: Pull the last N production inputs from your logging store (e.g., 10,000 requests) and their classified intent labels. (2) Baseline retrieval: Load the reference distribution from a versioned artifact store (S3, MLflow, or your model registry). The baseline should be a snapshot of intent distributions from a known-good evaluation period, stored as a JSON object mapping intent labels to normalized frequencies. (3) Prompt invocation: Format the current distribution and baseline distribution into the [CURRENT_DISTRIBUTION] and [BASELINE_DISTRIBUTION] placeholders. Set [DRIFT_THRESHOLD] to your organization's acceptable divergence (e.g., 0.05 for JS distance). Set [ALERT_SEVERITY_MAP] to define which drift magnitudes trigger info, warning, or critical alerts. (4) Response validation: Parse the model's JSON output and validate that flagged_intents contains only labels present in your taxonomy, that drift_score is a float between 0 and 1, and that recommended_action is one of your predefined enum values (none, monitor, retrain, investigate). If validation fails, retry once with a stricter schema reminder; if it fails again, log the raw response and escalate to the MLOps on-call channel. (5) Action dispatch: If recommended_action is retrain or investigate, the harness should automatically open a ticket in your issue tracker with the drift summary, flagged categories, and a link to the raw distribution data. Do not automatically trigger retraining—require human approval for any pipeline that modifies production model weights.

Model choice matters here. Use a model with strong JSON-following behavior and a context window large enough to hold both full distributions without truncation. GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are appropriate. Avoid models under 7B parameters for this task; they tend to hallucinate statistical comparisons or collapse the output structure. Set temperature=0 to maximize reproducibility. If your intent taxonomy exceeds 50 labels, consider bucketing low-frequency intents into an other category before sending distributions to the prompt, or split the comparison across multiple prompt calls by intent group. Log every drift assessment—including the raw prompt, model response, parsed JSON, and any validation errors—to your prompt observability tool (Langfuse, Braintrust, or a custom trace store) so you can debug false-positive drift alerts and refine thresholds over time.