Production Input Clustering for Drift Detection Prompt

This prompt is designed for data engineers and MLOps teams who need to preprocess raw production logs before running a full drift analysis. It takes a batch of recent production inputs and an existing few-shot example set, then outputs labeled clusters with representative samples, cluster sizes, and novelty scores. Use this as an upstream step to identify which production inputs are well-covered by current examples and which represent new, unmodeled behavior. This is not a drift detection prompt itself. It is a clustering and scoring step that feeds downstream coverage gap analysis, staleness scoring, and example refresh workflows.

The ideal user is someone who owns a production prompt pipeline and needs to answer the question: 'Has our traffic changed in ways our examples don't cover?' The prompt requires two inputs: a sample of recent production requests and the current few-shot example set used in the prompt. It returns structured cluster assignments, a novelty score per cluster indicating how far each group is from the existing examples, and representative samples you can inspect directly. Wire this into a scheduled job that runs before your drift scoring prompts, or trigger it ad-hoc when you suspect a distribution shift after a product change, user population shift, or seasonal event.

Do not use this prompt when you already have a dedicated embedding-based clustering pipeline or when you need real-time drift detection on individual requests. This prompt is batch-oriented and designed for periodic analysis, not per-request scoring. It also assumes your production inputs and examples are text-based and can be compared through semantic similarity. If your inputs are structured API calls with tool selections, use a schema-aware clustering approach instead. After running this prompt, feed the cluster assignments and novelty scores into your staleness scoring prompt or coverage gap analysis prompt to decide which examples need refreshing.

This template is the core instruction block for the Production Input Clustering for Drift Detection Prompt. It is designed to be dropped into an application harness that supplies the production input batch and the current example set. The prompt instructs the model to perform unsupervised clustering on the raw inputs, generate human-readable labels, and compute a novelty score for each cluster relative to the examples you already use in your prompt. The output is a structured JSON report that feeds directly into downstream staleness analysis and coverage gap detection workflows.

textCopy
You are an AI data engineer analyzing production inputs for drift detection.

Your task is to cluster the following raw production inputs, label each cluster, and score how novel each cluster is compared to the existing few-shot example set.

## INPUTS

### Production Input Batch
[PRODUCTION_INPUTS]

### Existing Few-Shot Example Set
[EXAMPLE_SET]

## INSTRUCTIONS

1. **Cluster the production inputs** into semantically meaningful groups. Each input belongs to exactly one cluster. If an input does not fit any cluster, place it in an "Unclustered" group.
2. **Label each cluster** with a short, descriptive name that captures the core intent, topic, or pattern shared by its members. Avoid vague labels like "Group 1" or "Miscellaneous."
3. **Select representative samples** for each cluster. Choose up to [MAX_SAMPLES_PER_CLUSTER] inputs that best illustrate the cluster's defining characteristics.
4. **Score novelty** for each cluster on a scale from 0.0 to 1.0, where:
   - 0.0 means the cluster is fully covered by existing examples (no novelty).
   - 1.0 means the cluster represents completely unseen patterns with no similar examples in the existing set.
   Base this score on semantic similarity between the cluster's inputs and the existing examples.
5. **Provide a brief justification** for each novelty score, citing specific differences or gaps.

## OUTPUT SCHEMA

Return a valid JSON object with this exact structure:
{
  "clusters": [
    {
      "cluster_id": "string",
      "label": "string",
      "size": integer,
      "percentage_of_total": float,
      "representative_samples": ["string"],
      "novelty_score": float,
      "novelty_justification": "string"
    }
  ],
  "unclustered_count": integer,
  "total_inputs_processed": integer,
  "summary": "string"
}

## CONSTRAINTS

- Do not invent inputs. Only cluster the provided production inputs.
- Do not modify the existing example set. Only compare against it.
- If the production input batch is empty, return an empty clusters array with a summary explaining the empty result.
- Novelty scores must be justified with specific evidence, not generic statements.
- Preserve the original text of representative samples exactly as they appear in the production inputs.

To adapt this template, replace [PRODUCTION_INPUTS] with a JSON array of raw user inputs or log entries from your production system. Replace [EXAMPLE_SET] with a JSON array of your current few-shot examples, each containing at minimum an input field. Set [MAX_SAMPLES_PER_CLUSTER] to a number that balances cluster readability against token budget—typically 3 to 5. If your production inputs contain PII or sensitive data, redact them before insertion or use a local model deployment. The output schema is intentionally flat to simplify downstream parsing; validate that the model returns valid JSON before passing results to your drift analysis pipeline. For high-stakes production monitoring, add a human review step for clusters with novelty scores above 0.7 before triggering automated example refresh workflows.

This prompt is designed as an upstream step in a drift detection pipeline, not a standalone analysis tool. Its job is to transform raw, unstructured production logs into structured cluster metadata that downstream prompts—such as coverage gap analyzers, staleness scorers, and refresh triggers—can consume programmatically. The harness must enforce a strict JSON output contract, validate cluster coherence, and log novelty scores for later threshold comparison. Because the prompt operates on production data that may contain PII or sensitive user inputs, the harness should apply redaction or pseudonymization before the data reaches the model, and the clustering output must never include raw user text unless explicitly approved for retention.

Wire the prompt into a batch processing job or a scheduled workflow (e.g., a daily cron or a step in an ML pipeline DAG). The harness should: (1) sample N production inputs from the target logging store (e.g., 500–2000 records, depending on cost and latency tolerance); (2) apply any PII redaction filters; (3) inject the sample and the existing example set into the prompt's [PRODUCTION_SAMPLE] and [EXISTING_EXAMPLE_SET] placeholders; (4) call the model with a low temperature (0.0–0.2) and a strict JSON mode or structured output API parameter; (5) parse the response and validate that every cluster object contains the required fields: cluster_id, label, size, representative_samples, and novelty_score. If validation fails, retry once with an explicit repair instruction appended to the prompt. Log the raw response, the parsed output, and any validation errors to an observability store for trace analysis.

After validation, the harness should write the structured cluster report to a downstream data store (e.g., a feature table, a vector database metadata table, or a drift analysis queue) where subsequent prompts can query it. Do not treat this prompt's output as a final alert or decision. It is a structured input for other prompts—such as Example Drift Severity Classification or Example Refresh Trigger Evaluation—that will apply thresholds and business logic. Avoid wiring this prompt directly to an alerting channel; instead, route its output through a decision layer that can suppress noise, compare against historical baselines, and require human approval before triggering automated example refreshes in production-critical systems.