Distractor Passage Identification Prompt

This prompt is for QA accuracy teams and RAG pipeline builders who need to identify retrieved passages that appear relevant on the surface but contain information that would mislead or corrupt answer generation. The core job is to flag 'distractor' passages—content that shares keywords or topical overlap with the query but introduces contradictory facts, outdated information, speculative claims, or contextually inappropriate details that would degrade the final answer if included. The ideal user is an AI engineer or QA lead who already has a retrieval pipeline producing candidate passages and needs a programmatic filter before those passages reach the answer synthesis step.

Use this prompt when your retrieval system returns passages that pass basic relevance thresholds but still produce hallucinated or conflicting answers downstream. This is common in large document collections where outdated versions coexist with current ones, where speculative or opinionated content shares vocabulary with factual content, or where domain-specific jargon creates false semantic matches. The prompt expects a query and a set of retrieved passages as input, and it outputs a structured identification of distractors with the specific misleading element flagged and a recommended exclusion decision. Do not use this prompt as a general relevance filter—it is specifically designed to catch passages that look relevant but are harmful to answer quality, not passages that are simply off-topic.

This prompt is not a substitute for retrieval quality improvements or embedding model tuning. If your retrieval system consistently returns distractors at high rates, investigate your indexing strategy, chunking parameters, and embedding model choice first. This prompt works best as a safety net in a multi-stage filtering pipeline, placed after relevance filtering and before answer generation. For high-stakes domains such as clinical, legal, or financial Q&A, always pair this prompt's output with human review of flagged distractors before excluding them from context, and log exclusion decisions for audit trails.

The Distractor Passage Identification Prompt is designed to sit between retrieval and answer generation in a RAG pipeline. Its job is to catch passages that score well on embedding similarity or keyword overlap but contain information that would actively mislead the model if used for synthesis. This is a higher bar than simple relevance filtering—distractors are passages that look right but point toward wrong answers. Use this prompt when your retrieval set contains content from multiple sources, versions, or perspectives where plausible-sounding misinformation can slip through.

textCopy
You are a context quality auditor for a question-answering system. Your task is to examine a set of retrieved passages and identify any that are DISTRACTORS—passages that appear relevant to the query but contain information that would mislead answer generation if used.

## INPUT

**Query:** [QUERY]

**Retrieved Passages:**
[PASSAGES]

## DEFINITION OF A DISTRACTOR

A distractor passage meets ALL of these criteria:
1. It has surface-level relevance to the query (shared keywords, entities, or topic overlap).
2. It contains factual information that contradicts the correct answer, introduces a common misconception, or describes a related but different concept that could be confused with the answer.
3. Including this passage in context would likely cause the answer generator to produce an incorrect, misleading, or conflated response.

## WHAT IS NOT A DISTRACTOR

Do NOT flag passages that are:
- Simply irrelevant or off-topic (those are noise, not distractors).
- Partially relevant but incomplete (those need supplementation, not removal).
- Correct but about a different aspect of the query (those may still be useful).
- Redundant with other passages (those are duplicates, not distractors).

## OUTPUT SCHEMA

Return a JSON object with this exact structure:

{
  "distractors_found": boolean,
  "distractor_count": number,
  "distractors": [
    {
      "passage_id": "string (the identifier from the input)",
      "misleading_element": "string (the specific claim, conflation, or misconception that makes this passage dangerous)",
      "exclusion_recommendation": "string (one of: 'exclude', 'exclude_and_warn', 'quarantine_for_review')",
      "severity": "string (one of: 'high', 'medium', 'low')",
      "explanation": "string (why this passage would mislead answer generation)"
    }
  ],
  "safe_passage_ids": ["list of passage IDs that are safe to use"],
  "summary": "string (brief summary of findings and recommended action)"
}

## CONSTRAINTS

- Only flag passages that would actively mislead, not merely unhelpful ones.
- For each distractor, identify the SPECIFIC misleading element—do not give vague reasons.
- If no distractors are found, return an empty distractors array and set distractor_count to 0.
- Preserve all input passage IDs exactly as provided.
- If you are uncertain whether a passage is a distractor, set exclusion_recommendation to 'quarantine_for_review' and severity to 'low'.

## EXAMPLES

**Example 1: Clear Distractor**
Query: "When was the Python programming language first released?"
Passage: "Python 3.0 was released in December 2008 and introduced many backward-incompatible changes."
Assessment: DISTRACTOR. This passage is about Python 3.0, not the initial Python release (1991). A model might incorrectly answer 2008.

**Example 2: Not a Distractor**
Query: "When was the Python programming language first released?"
Passage: "Python was created by Guido van Rossum at CWI in the Netherlands."
Assessment: NOT A DISTRACTOR. This is relevant background information that does not contradict the correct answer.

**Example 3: Subtle Distractor**
Query: "What causes seasons on Earth?"
Passage: "The Earth's elliptical orbit means it is closer to the sun in January than in July, causing temperature variations throughout the year."
Assessment: DISTRACTOR. This describes orbital distance effects, which is a common misconception. The correct answer is axial tilt. Including this would reinforce the misconception.

Adapt this template by adjusting the distractor definition to match your domain's specific failure modes. For legal documents, distractors might be superseded clauses that look current. For medical content, they might be drug interactions described for a different condition. For technical documentation, they might be deprecated API methods that still appear in search results. The severity levels and exclusion recommendations should map to your downstream handling logic—'quarantine_for_review' is appropriate when you have a human review queue, while 'exclude' works for fully automated pipelines with strong eval coverage.

The Distractor Passage Identification Prompt is designed to sit between retrieval and answer generation in a RAG pipeline. It receives a set of retrieved passages and the user query, then flags passages that appear relevant but contain misleading information. The implementation harness must treat this prompt as a filtering gate: passages flagged as distractors should be excluded from the context window before answer synthesis, while the distractor metadata (misleading element, reason) should be logged for pipeline observability and downstream eval.

Wire the prompt into your application as a pre-generation filter step. After retrieval returns k passages, batch them with the query and send them to the model. Parse the structured output—expect a JSON array of distractor objects, each with a passage_id, misleading_element, and exclusion_recommendation. Validate the output schema strictly: if passage_id references a passage not in the input set, discard that entry and log a schema violation. If the model returns malformed JSON, retry once with a stricter output constraint appended to the prompt. After two failures, fall back to including all passages and flag the incident for review. For high-stakes domains (legal, clinical, financial), route any passage where exclusion_recommendation is true to a human review queue before final exclusion, especially when the distractor count exceeds 30% of retrieved passages—this may indicate a retrieval quality problem rather than individual distractor passages.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller models for this task unless you've validated distractor recall against a golden dataset. Implement eval logging: for every invocation, record the query, the number of passages flagged, the specific misleading elements identified, and whether the final answer generation step produced a hallucination (detected via a separate hallucination check). Over time, this log becomes your distractor detection precision/recall dataset. Set up a periodic review cadence—sample 50 filtered passages per week, have a domain expert verify whether the distractor flags were correct, and tune your prompt or threshold based on false positive rates. The goal is to catch misleading passages without over-filtering and starving the answer generator of useful context.