Example Drift Detection Prompt for Production Classifiers

This playbook is for MLOps engineers and AI platform teams who maintain production classification systems that rely on few-shot examples. The core job-to-be-done is not classifying a new input, but auditing the health of the examples that teach the model how to classify. Use this prompt when you suspect silent accuracy degradation, when new user behaviors or product features have emerged since the example set was last curated, or when you need statistical evidence to justify an example refresh cycle to your team. The ideal user has access to a batch of recent production inputs and their corresponding model predictions, and they need to know whether the ground truth they are providing via examples has drifted from reality.

This prompt is a meta-analysis tool. It compares the semantic distribution of a batch of recent production inputs against the current set of few-shot examples to detect three failure modes: semantic drift (new topics or intents not represented in examples), distributional shift (the relative frequency of known categories has changed), and staleness (examples reference outdated product names, features, or policies). The output is not a new classifier; it is a diagnostic report containing drift scores, a list of affected categories, and actionable refresh recommendations. You should wire this into a scheduled monitoring job, not a real-time classification path. Run it weekly, after a product launch, or when your eval dashboard shows a drop in classification accuracy without an obvious cause.

Do not use this prompt as a substitute for standard model evaluation. It does not measure accuracy, precision, or recall directly. It measures the alignment between your example set and your production data distribution. If your production data is noisy or your example set was poorly constructed to begin with, this prompt will surface those issues, but it won't fix them. Also, avoid running this on very small batches of production inputs (fewer than 50-100 samples), as the statistical signal will be too weak to produce reliable drift scores. The next step after reading this section is to prepare a representative batch of recent production inputs and your current few-shot example set, then proceed to the prompt template to run the analysis.

This prompt template is designed to be integrated into an MLOps monitoring pipeline. It expects a JSON array of recent production inputs and a JSON object mapping each classification category to its representative few-shot examples. The model will analyze the semantic and distributional alignment between the two sets and return a structured drift analysis. Use this when you suspect your classifier's accuracy is degrading but haven't yet identified which categories are affected or whether your examples have gone stale.

textCopy
You are a production ML drift detector. Your task is to compare a set of recent production inputs against the few-shot examples used in a classifier's prompt. Determine whether the examples still represent the production data distribution.

## INPUT
- Production inputs: [PRODUCTION_INPUTS]
- Few-shot examples by category: [FEWSHOT_EXAMPLES]

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "drift_detected": boolean,
  "overall_drift_score": number (0.0 to 1.0),
  "category_analysis": [
    {
      "category": string,
      "drift_score": number (0.0 to 1.0),
      "staleness_flag": boolean,
      "affected_input_count": number,
      "evidence": [
        {
          "production_sample": string,
          "mismatch_reason": string
        }
      ],
      "recommendation": "refresh" | "monitor" | "ok"
    }
  ],
  "global_recommendation": string,
  "categories_needing_immediate_refresh": [string]
}

## CONSTRAINTS
- Only flag drift when you have statistical evidence from the provided samples.
- If a category has fewer than 3 production inputs, set drift_score to null and recommendation to "monitor".
- Base staleness on semantic shift, vocabulary change, or topic distribution mismatch, not on superficial wording differences.
- Do not hallucinate production inputs. Only reference samples present in [PRODUCTION_INPUTS].

To adapt this template, replace [PRODUCTION_INPUTS] with a JSON array of strings sampled from your production traffic over a recent time window (e.g., the last 1,000 requests). Replace [FEWSHOT_EXAMPLES] with a JSON object where each key is a category label and each value is an array of example strings currently used in your classifier's prompt. Before deploying, validate the output against a set of known drift scenarios to ensure the model correctly identifies semantic shifts rather than flagging normal vocabulary variation. For high-stakes classification systems, route any recommendation: "refresh" result to a human reviewer before updating production prompts.

This prompt is designed to run as a scheduled batch job within your existing MLOps infrastructure, not as a real-time request. The implementation harness must treat the prompt as a statistical monitoring probe that compares a sample of recent production inputs against the few-shot examples currently deployed in your classifier prompt. The core integration points are: a data sampler that pulls recent inputs from your inference logs, a prompt execution layer that invokes the drift detection prompt with both the production sample and the reference examples, and a post-processing validator that enforces the expected output schema before the results are written to your monitoring dashboard or alerting system. The harness should never modify the production classifier prompt directly; it only produces a drift report that a human or an automated approval workflow can act on.

Wire the prompt into a job runner (e.g., a cron-triggered Cloud Function, an Airflow DAG, or a Modal app) that executes the following steps: (1) Sample extraction — pull the last N inference requests from your logging store (e.g., BigQuery, Datadog, or an internal event stream), stratified by predicted class if possible, to ensure minority categories are represented. (2) Prompt assembly — inject the sampled inputs and the current few-shot examples into the [PRODUCTION_SAMPLE] and [REFERENCE_EXAMPLES] placeholders respectively. (3) Model invocation — call your chosen model with temperature=0 and a strict response_format constraint matching the [OUTPUT_SCHEMA] (a JSON object containing drift_score, affected_categories, recommendation, and evidence_summary). (4) Validation — run the raw model output through a JSON schema validator. If validation fails, retry once with the error message appended as a repair instruction. If it fails again, log the failure and escalate to the on-call channel rather than silently swallowing the alert. (5) Logging and storage — write the validated drift report to your monitoring database with a timestamp, the prompt version hash, and the sample size used. This creates an auditable trail for governance reviews.

Set your alerting thresholds based on the drift_score field (a float from 0.0 to 1.0). A score above 0.7 should trigger a human-review ticket in your project management system with the full drift report attached. Scores between 0.4 and 0.7 should generate a warning notification in your monitoring channel. Below 0.4, log the result silently for trend analysis. The affected_categories array tells you exactly which classes are drifting, so your refresh workflow can target only those examples rather than rebuilding the entire set. Critical constraint: never automatically swap production examples based on this prompt's output. The prompt identifies drift; it does not curate replacements. A separate curation prompt or human review step must produce the new examples. Finally, monitor the drift detection prompt itself — if it begins producing malformed outputs or contradictory scores, your monitoring pipeline has a blind spot that needs immediate attention.