Multi-Passage Deduplication and Selection Prompt

This prompt is designed for RAG pipeline builders who face a common retrieval failure mode: the top-k results from vector or keyword search contain near-duplicate passages that waste context window space and dilute the answer signal. The job-to-be-done is to transform a noisy, redundant retrieval set into a minimal set of unique, information-dense passages before they reach the answer synthesis step. The ideal user is an AI engineer or search engineer who already has a working retrieval pipeline and needs a deduplication layer that preserves complementary details—such as a passage that adds a date or a specific constraint—while removing verbatim or high-overlap copies. You should use this prompt when your retrieval corpus contains multiple versions of the same document, when chunking strategies produce overlapping passages, or when you observe that your synthesis model is wasting tokens on repeated evidence and producing less precise answers as a result.

You should not use this prompt when your retrieval set is already small (fewer than 3 passages), when passages are guaranteed unique by your chunking strategy, or when your downstream model has a context window large enough to absorb redundancy without quality degradation. This prompt is also not a substitute for better retrieval. If your search pipeline consistently returns irrelevant results, fix retrieval quality first—deduplication cannot rescue bad recall. The prompt assumes that input passages are at least partially relevant to the query and that the primary problem is redundancy, not noise. For noise filtering, pair this with an evidence relevance scoring step before deduplication. The prompt requires a user query and a list of candidate passages as input, and it produces a deduplicated subset with information gain scores and explicit exclusion reasons. This structured output is designed to be consumed programmatically by a downstream synthesis prompt or logged for observability.

Before deploying this prompt, define your semantic similarity threshold. The prompt uses an information gain scoring approach rather than strict cosine similarity, which means it will preserve passages that share a topic but contribute distinct facts. If your use case requires aggressive deduplication (e.g., removing all passages about the same entity regardless of new details), you will need to adjust the instructions or add a stricter similarity constraint. For high-stakes domains such as clinical or legal RAG, always include a human review step or an automated eval that verifies no unique evidence was incorrectly removed. The next section provides the copy-ready prompt template with placeholders you can adapt to your retrieval schema and output format requirements.

This prompt template is designed to be dropped directly into your RAG pipeline after retrieval and before answer synthesis. It accepts a list of retrieved passages that may contain semantic near-duplicates, overlapping content, or redundant information, and returns a deduplicated set of passages ranked by unique information contribution. The template uses square-bracket placeholders that you replace with your actual retrieval output, deduplication thresholds, and output format requirements. Copy the template below, replace the placeholders with your pipeline's values, and wire it into your evidence selection stage.

textCopy
You are an evidence deduplication and selection system. Your job is to process a set of retrieved passages, identify semantic near-duplicates and overlapping content, and return a deduplicated set of unique passages ranked by information gain.

## INPUT PASSAGES
[RETRIEVED_PASSAGES]

## DEDUPLICATION RULES
1. Compare all passages pairwise for semantic overlap. Two passages are near-duplicates if they convey substantially the same information, even if wording differs.
2. When a group of near-duplicate passages is found, select the single passage that is most complete, precise, and well-sourced. Discard the others.
3. For partially overlapping passages, keep both only if each contributes unique information not present in the other. If one passage subsumes the other entirely, keep only the more complete passage.
4. Do not discard a passage solely because it is shorter. Short passages with unique facts are valuable.
5. Do not merge or rewrite passages. Always return original passage text.

## INFORMATION GAIN SCORING
For each kept passage, assign an information gain score from 0.0 to 1.0 based on:
- **Novelty**: How much new information this passage adds beyond what earlier-ranked passages already cover.
- **Specificity**: How precise and fact-dense the passage is, versus vague or generic statements.
- **Relevance**: How directly the passage addresses the user query: [USER_QUERY]

## OUTPUT FORMAT
Return a JSON object with this exact schema:
{
  "deduplicated_passages": [
    {
      "passage_id": "original ID from input",
      "passage_text": "exact text of the selected passage",
      "information_gain_score": 0.85,
      "kept_because": "brief explanation of unique contribution",
      "duplicates_removed": ["passage_id_1", "passage_id_2"]
    }
  ],
  "total_input_passages": 0,
  "total_kept_passages": 0,
  "total_duplicates_removed": 0
}

## CONSTRAINTS
- [SIMILARITY_THRESHOLD]: Treat passages as near-duplicates if semantic overlap exceeds this threshold (e.g., 0.85).
- [MAX_OUTPUT_PASSAGES]: Maximum number of passages to return. If more unique passages exist, keep the highest-scoring ones.
- [MIN_INFORMATION_GAIN]: Exclude any passage scoring below this threshold, even if unique.

To adapt this template for your pipeline, replace [RETRIEVED_PASSAGES] with your actual retrieval output, formatted as an array of objects each containing a passage_id and passage_text. Set [USER_QUERY] to the original question that triggered retrieval. Tune [SIMILARITY_THRESHOLD] based on your tolerance for near-duplicates—start at 0.85 and lower it if you see redundant passages slipping through. Set [MAX_OUTPUT_PASSAGES] to match your downstream context window budget, and use [MIN_INFORMATION_GAIN] to filter out low-signal passages that add noise rather than value. If your retrieval system does not provide passage IDs, generate them from chunk indices or content hashes before calling this prompt. For high-stakes domains, add a human review step before the deduplicated set feeds into answer synthesis, and log the duplicates_removed field to maintain an audit trail of what was excluded and why.

The Multi-Passage Deduplication and Selection Prompt is designed to sit between retrieval and answer synthesis in a RAG pipeline. It accepts a list of retrieved passages and returns a deduplicated set with information gain scores. In production, this prompt is not a one-shot call—it requires a harness that validates the output schema, handles malformed responses, logs selection decisions, and respects context window budgets. The harness should treat the prompt as a deterministic filter: same inputs should produce a stable deduplicated set, and any deviation should be logged as a regression signal.

Wire the prompt into your application as a post-retrieval processing step. After your retriever returns N passages, assemble the prompt with [PASSAGES] as a JSON array of objects, each containing id, text, and optional source and date fields. Set [SIMILARITY_THRESHOLD] to a float between 0.0 and 1.0 (start at 0.85 for near-duplicate detection). Set [MAX_OUTPUT_PASSAGES] to your context window budget divided by average passage tokens. Call the model with response_format set to JSON mode or structured outputs if available. Parse the response into a DeduplicationResult schema with fields: selected_passages (array of passage objects with added information_gain_score and uniqueness_rationale), removed_passages (array of IDs with removal_reason and duplicate_of_id), and coverage_summary (string describing what information is preserved). Validate that every duplicate_of_id in removed_passages exists in selected_passages, that no passage ID appears in both arrays, and that information_gain_score values are floats between 0.0 and 1.0. If validation fails, retry once with the validation errors appended to [CONSTRAINTS]. If the retry also fails, log the failure, fall back to a simpler deduplication method (e.g., exact text hash matching), and alert the on-call channel.

For observability, log the full deduplication trace: input passage count, output passage count, removed IDs with reasons, and the coverage_summary string. Compare these metrics across prompt versions to detect regressions in deduplication behavior. For high-stakes domains where information loss from over-deduplication could cause harm, add a human review step when the deduplication ratio exceeds a threshold (e.g., more than 50% of passages removed) or when information_gain_score values cluster near 0.5, indicating ambiguity. Use a smaller, faster model for this prompt when latency matters—deduplication is a classification-style task that does not require the largest model. Test with golden datasets containing known near-duplicate pairs, exact duplicates, and complementary passages to calibrate your [SIMILARITY_THRESHOLD] before production deployment.