Topic Drift Detection Prompt for Retrieved Passages

This prompt is for retrieval quality engineers and RAG pipeline builders who need to catch a specific, high-cost failure mode: passages that score well on embedding similarity but are semantically off-topic for the user's actual question. The job-to-be-done is not general relevance filtering, but targeted semantic drift detection. You use this prompt when your vector store returns a set of candidate passages, and you need to programmatically identify and flag any passage that has drifted onto a different subject, entity, or intent despite surface-level keyword or embedding overlap. The ideal user is someone who already has retrieval metrics in place and is now debugging precision failures that standard similarity scores miss.

You should reach for this prompt when you observe answers that are factually correct in isolation but wrong for the query—for example, a query about 'Apple's M3 chip performance' returning a highly-ranked passage about 'Apple's Q3 financial performance.' The prompt is designed to operate as a post-retrieval filter, sitting between your retriever and your answer generation step. It expects a specific query and a list of retrieved passages as input, and it outputs a structured drift assessment for each passage, identifying the off-topic element. This is not a general-purpose relevance classifier; it is a precision tool for catching semantic drift that embedding models miss.

Do not use this prompt as your primary or only relevance filter. It is too narrow and too expensive for broad filtering. It is a diagnostic and precision-enhancement tool. Do not use it when your retrieval pipeline is already producing high-precision results, or when the cost of an extra LLM call per passage outweighs the risk of a drifted answer. Avoid using it on very short, ambiguous queries where 'topic' itself is ill-defined. In high-stakes domains like healthcare or finance, the output of this prompt should be treated as a strong signal for human review or automatic exclusion, but never as the sole gatekeeper for answer generation. The next step after reading this section is to examine the prompt template and adapt its output schema to your logging and filtering infrastructure.

This prompt template is designed to be copied directly into your application code or prompt management system. It instructs the model to analyze a set of retrieved passages against a user query and identify any that have drifted off-topic despite high embedding similarity. The template uses square-bracket placeholders for all variable inputs, making it straightforward to wire into a RAG pipeline where the query and retrieved context are injected at runtime.

textCopy
You are a retrieval quality auditor. Your task is to analyze a set of retrieved passages and identify any that exhibit topic drift relative to the user's query.

## INPUT
**User Query:** [QUERY]
**Retrieved Passages:** [PASSAGES]

## TASK
For each passage, determine whether it is on-topic or off-topic relative to the query. A passage is off-topic if it discusses a different subject, answers a different question, or focuses on an aspect unrelated to the user's intent, even if it shares keywords or surface-level similarity with the query.

For each off-topic passage, identify:
1. The specific element that has drifted (e.g., subject entity, time period, action, domain, or question type).
2. A brief explanation of why the passage is not relevant.

## OUTPUT_SCHEMA
Return a JSON object with the following structure:
```json
{
  "query": "string (the original query)",
  "total_passages": integer,
  "on_topic_count": integer,
  "off_topic_count": integer,
  "passages": [
    {
      "passage_id": "string or integer",
      "passage_text": "string (the full passage text)",
      "is_on_topic": boolean,
      "drift_element": "string | null (the specific off-topic element, or null if on-topic)",
      "drift_explanation": "string | null (brief explanation of the drift, or null if on-topic)"
    }
  ]
}

CONSTRAINTS

• Do not mark a passage as off-topic simply because it lacks sufficient detail to fully answer the query. Topic drift is about subject mismatch, not completeness.
• If a passage is partially relevant but contains some off-topic content, mark it as on-topic if the primary focus aligns with the query.
• Be precise in identifying the drift element. Use categories such as: subject entity, time period, geographic location, action or event, domain or field, question type, or sentiment mismatch.
• If all passages are on-topic, return an empty list for off-topic passages and set off_topic_count to 0.

EXAMPLES

Example 1: Clear Topic Drift Query: "What are the side effects of metformin?" Passage: "Metformin was first synthesized in 1922 and received FDA approval in 1994. The drug's development history includes several formulation improvements." Drift Element: time period and focus (history vs. side effects) Explanation: The passage discusses the drug's development history, not its side effects.

Example 2: No Drift Query: "How do I reset my router password?" Passage: "To reset your router password, locate the reset button on the back of the device. Press and hold for 10 seconds until the lights flash." Drift Element: null Explanation: null

To adapt this template for your own use, replace the [QUERY] and [PASSAGES] placeholders with your actual data. The passages should be provided as a JSON array of objects, each with at least an id and text field. If your retrieval system uses different identifiers, adjust the passage_id field in the output schema accordingly. For high-stakes applications where topic drift could lead to misleading answers, consider adding a [RISK_LEVEL] parameter that adjusts the strictness of drift detection—for example, flagging borderline cases as off-topic when risk is high.

Before deploying this prompt in production, validate its behavior against a golden dataset of known on-topic and off-topic passage-query pairs. Pay special attention to borderline cases where passages share keywords with the query but address a different intent. If your application operates in a regulated domain, always include a human review step for flagged off-topic passages before they are excluded from answer generation, as over-filtering can remove context that a human would recognize as relevant.

The topic drift detection prompt operates as a post-retrieval filter, sitting between your vector store and answer generation. For each retrieved passage, call this prompt to classify whether the passage has drifted off the query topic despite high embedding similarity. The prompt expects two inputs: the original user query and a single retrieved passage. It returns a structured classification with a drift flag, the specific off-topic element identified, and a confidence score. Wire this as a batch operation—process all retrieved passages in parallel before they reach the context assembly step, discarding or deprioritizing passages flagged as drifted.

Implement a validation layer that checks the structured output before acting on it. The response must contain a drift_detected boolean, an off_topic_element string (empty if no drift), and a confidence float between 0.0 and 1.0. If the model returns malformed JSON or missing fields, retry once with the same inputs and a stronger format constraint appended to the prompt. Log all validation failures and retry outcomes. For high-stakes pipelines, route passages with confidence scores in the ambiguous range (0.4–0.6) to a human review queue rather than auto-filtering. Set a drift threshold based on your tolerance for false positives—starting at confidence >= 0.7 with drift_detected == true is a reasonable default that you should tune against a labeled eval set.

Model choice matters here. This is a classification task that benefits from instruction-following precision over creative fluency. GPT-4o, Claude 3.5 Sonnet, and similarly capable models handle this well. Avoid smaller or older models that may struggle with the nuanced distinction between semantic similarity and topical relevance. If latency is critical, consider running this check on only the top-N retrieved passages rather than the full retrieval set. Instrument the pipeline to emit metrics: drift rate per query, average confidence, validation failure rate, and latency per passage. These metrics feed your retrieval quality dashboards and help detect embedding model drift or index staleness over time.

Do not use this prompt as a standalone quality gate without upstream retrieval hygiene. If your vector store consistently returns off-topic passages with high similarity scores, the root cause is likely in your embedding model, chunking strategy, or index configuration—not something a downstream filter should silently absorb. Use the drift detection output to flag problematic queries and passages for retrieval tuning. Store the off_topic_element field in your observability traces so you can cluster failure patterns and identify which topics or query types trigger the most semantic drift.