Unseen Intent Cluster Escalation Prompt

Conversational AI systems degrade silently when users ask for things the system was never designed to handle. The existing intent taxonomy captures what the product team planned for, but production traffic always contains requests that fall outside those boundaries. This prompt identifies user inputs that do not fit any known intent, clusters similar unknowns into coherent groups, and escalates representative examples with diagnostic metadata so the taxonomy team can decide whether to add new intents, merge clusters, or treat them as noise. Use this prompt when you need structured visibility into intent drift before it becomes a support backlog or a retention problem.

This prompt is designed for offline or near-real-time batch analysis of logged utterances, not for live traffic routing. Run it against a daily or weekly sample of unhandled utterances—typically those that fell into a catch-all fallback intent or received low confidence scores from your production classifier. The prompt expects a list of utterances, your current intent taxonomy with definitions, and any existing fallback or out-of-scope labels. It returns clustered groups of unseen intents, each with a proposed label, representative examples, frequency counts, and a diagnostic assessment of whether the cluster represents a genuine taxonomy gap, a sub-intent of an existing category, or noise that should remain unmodeled.

Do not use this prompt for real-time routing decisions or as a replacement for a production intent classifier. It is not latency-optimized and does not guarantee deterministic cluster assignments across runs. For live traffic, continue routing unknown intents to a fallback handler or human agent. This prompt's output should feed into a taxonomy review cadence—weekly or biweekly sessions where product and conversation design teams review escalated clusters, decide on taxonomy updates, and measure drift trends over time. Pair this prompt with a logging pipeline that captures raw utterances, classifier confidence scores, and any user feedback signals (e.g., abandonment, negative ratings) to prioritize clusters that correlate with poor user outcomes.

This prompt is designed to be inserted into a batch processing or streaming pipeline that handles utterances flagged as 'unclassified' or 'low confidence' by your primary intent classifier. Its job is not to classify the utterance into an existing intent, but to group similar novel utterances together, identify representative examples for each group, and produce a structured escalation report that a product or conversation design team can use to update the intent taxonomy. The prompt assumes you have already filtered for utterances that fall below your confidence threshold and have collected them over a defined time window.

textCopy
You are an intent taxonomy analyst reviewing unclassified user utterances from a production conversational AI system. Your task is to identify clusters of semantically similar utterances that do not fit any existing intent, and prepare a structured escalation report for the taxonomy update team.

## INPUT
A list of unclassified utterances collected during the [TIME_WINDOW] window. Each utterance includes a timestamp and the raw user text.

[UTTERANCE_LIST]

## EXISTING TAXONOMY
For reference, here are the currently defined intents. Do not force any utterance into these categories.

[EXISTING_INTENT_TAXONOMY]

## INSTRUCTIONS
1. Review all utterances and group them into semantic clusters based on shared user goals, not just keyword overlap.
2. For each cluster, assign a provisional label that describes the underlying user need (e.g., "request_bulk_data_export", "complain_about_integration_latency").
3. Select 3-5 representative examples per cluster that best illustrate the range of phrasing within that intent.
4. For each cluster, estimate the volume (count of utterances) and note any temporal patterns (e.g., spike on Tuesday, steady growth over the week).
5. Identify any utterances that appear to be genuine one-off noise, not part of any cluster, and flag them separately.
6. If any cluster suggests a high-risk or urgent user need (e.g., data deletion requests, account security concerns, billing disputes), mark it with [PRIORITY: HIGH] and explain why.

## OUTPUT FORMAT
Return a valid JSON object with the following schema:
{
  "report_id": "string",
  "time_window": "string",
  "total_utterances_processed": number,
  "clusters": [
    {
      "provisional_label": "string",
      "description": "string",
      "utterance_count": number,
      "temporal_notes": "string | null",
      "priority": "HIGH | MEDIUM | LOW",
      "priority_rationale": "string | null",
      "representative_examples": ["string"],
      "suggested_taxonomy_placement": "string"
    }
  ],
  "unclustered_noise": [
    {
      "utterance": "string",
      "reason_for_exclusion": "string"
    }
  ]
}

## CONSTRAINTS
- Do not invent intents that match the existing taxonomy. If an utterance fits an existing intent, it should not be in this input.
- Do not merge clusters that represent distinct user goals, even if they share vocabulary.
- If fewer than [MIN_CLUSTER_SIZE] utterances share a goal, treat them as noise unless they represent a high-risk need.
- Be conservative with priority escalation. Only mark HIGH if the user need implies financial, security, regulatory, or significant churn risk.

To adapt this prompt, replace the square-bracket placeholders with your pipeline's runtime values. [UTTERANCE_LIST] should be a JSON array of objects with at least id, text, and timestamp fields. [EXISTING_INTENT_TAXONOMY] should be a list of intent names and brief descriptions so the model can avoid re-classifying known intents. [TIME_WINDOW] is a human-readable label for the collection period. [MIN_CLUSTER_SIZE] controls the sensitivity of cluster formation—set it to 3-5 for production use. Before deploying, run this prompt against a labeled dataset of known drift events and measure cluster coherence (e.g., using normalized mutual information against human-labeled clusters) and coverage (recall of known novel intents). If the output JSON fails schema validation, implement a retry with the validation error message injected into [CONSTRAINTS] rather than silently accepting malformed escalation payloads.

The Unseen Intent Cluster Escalation Prompt is designed to operate as a periodic batch job or a streaming window processor, not as a real-time classifier on every user utterance. In production, you will typically run this prompt against a rolling window of low-confidence or unclassified utterances—often the tail of inputs that fall below your primary intent classifier's confidence threshold. The prompt expects a batch of utterances with their existing classifier scores and metadata, and it returns proposed clusters, representative examples, and an escalation recommendation. Wire this into your pipeline by first aggregating inputs from your intent classifier's rejection path (e.g., all utterances with confidence < 0.6 or intent = 'unknown') over a configurable time window—daily, hourly, or per N utterances—and then submitting that batch to the prompt.

Before calling the model, validate the input batch against a minimum size threshold (fewer than 10 utterances rarely produces meaningful clusters) and a maximum token budget (truncate or sample if the batch exceeds your model's context window). After receiving the model's structured output, run a programmatic validator that checks: (1) every proposed cluster has at least the minimum required representative examples, (2) cluster labels are distinct and non-overlapping, (3) the escalation_recommended field is a boolean, and (4) the coverage_ratio is a float between 0 and 1. If validation fails, retry once with the validation errors appended to the prompt as feedback. If the retry also fails, log the failure and escalate the raw batch to the human review queue with a note that automated clustering was unsuccessful. For model choice, use a model with strong semantic comparison capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent) since the task requires nuanced similarity judgment across utterances. Avoid smaller or older models that may collapse distinct clusters or miss subtle intent boundaries.

Log every clustering run with: the batch ID, timestamp, number of input utterances, number of clusters produced, cluster sizes, coverage ratio, and whether escalation was recommended. Store the representative examples per cluster for later taxonomy updates. When escalation_recommended is true, route the full clustering output to your human review queue with a structured payload that includes the cluster summaries, representative examples, and a link to the raw batch. The human reviewer should be able to accept clusters as new intents, merge clusters, reject noise, or request re-clustering with different parameters. Avoid wiring this prompt directly into automatic intent taxonomy updates—always require human approval before adding new intents to your production classifier, since incorrect clusters can degrade downstream routing and analytics. For eval, periodically compare the prompt's cluster assignments against human-labeled clusters on the same batch and track cluster homogeneity scores and escalation recall against known drift events.