Evidence Deduplication Prompt to Reclaim Tokens

This prompt is designed for the context assembly layer of a RAG pipeline, specifically the moment after retrieval and before final answer generation. Its job-to-be-done is to reclaim context window space by identifying and merging semantically redundant passages that were retrieved from dense vector stores, keyword indexes, or hybrid search systems. The ideal user is an AI engineer or RAG system builder who observes that their retrieved context blocks contain overlapping or near-identical information, leading to wasted tokens, increased latency, and diluted model attention. You should use this prompt when your retrieval system returns multiple chunks that say essentially the same thing in different words, when you need to maximize evidence density per token, and when you require a human-readable merge report to audit what was consolidated.

Do not use this prompt when the redundancy problem can be solved upstream with better retrieval parameters. If your vector store is returning exact duplicate documents, fix the ingestion deduplication logic first. If your hybrid search is returning the same passage from both vector and keyword indexes, add a simple exact-match or hash-based deduplication step before the LLM call—it is cheaper and faster. This prompt is also inappropriate when passages contain complementary details that appear similar but are factually distinct; merging them would lose critical nuance. Reserve this prompt for cases where semantic judgment is required to decide whether two passages are true duplicates, near-duplicates with minor wording differences, or distinct pieces of evidence that happen to share vocabulary. The prompt belongs in the context assembly layer precisely because it requires the model's semantic understanding to make these distinctions.

Before deploying this prompt, establish a baseline: measure your current context window utilization, average tokens per evidence block, and answer quality metrics. After integrating the deduplication step, compare token savings against any change in answer completeness or factual accuracy. The prompt includes a token savings report in its output schema, which you should log and monitor over time. If you observe that deduplication is consistently saving fewer than 10% of tokens, reconsider whether the upstream retrieval deduplication heuristics would be more cost-effective. If you observe answer quality degradation after deduplication, check whether the model is over-merging distinct passages—this is the most common failure mode and requires tuning the deduplication threshold in the prompt's constraints.

The prompt below is designed to be copied directly into your prompt assembly pipeline. It takes a set of retrieved passages, identifies semantic duplicates and near-duplicates, merges them into a single consolidated evidence block, and returns a structured report. The report includes the deduplicated evidence set, a merge log explaining which passages were combined, and a token savings summary. All placeholders are enclosed in square brackets. Replace each placeholder with your actual data and configuration before sending the prompt to the model.

textCopy
You are an evidence deduplication engine operating inside a RAG pipeline. Your task is to process a set of retrieved passages, identify duplicates and near-duplicates, merge them into a single consolidated evidence set, and report your work.

## INPUT

**Retrieved Passages:**
[RETRIEVED_PASSAGES]

**Deduplication Threshold:** [SIMILARITY_THRESHOLD]
**Merge Strategy:** [MERGE_STRATEGY]
**Maximum Merged Passage Length (tokens):** [MAX_MERGED_TOKENS]

## TASK

1. Compare all passages pairwise and identify groups that convey the same factual content.
2. For each group of duplicates or near-duplicates, produce a single merged passage that:
   - Preserves all unique facts from the group.
   - Resolves minor contradictions by noting the conflict.
   - Drops redundant phrasing.
   - Stays within the maximum merged passage length.
3. Passages with no duplicates must be included unchanged.
4. Count the original tokens and the deduplicated tokens.

## OUTPUT SCHEMA

Return a valid JSON object with exactly this structure:

{
  "deduplicated_evidence": [
    {
      "passage_id": "merged_1",
      "text": "Consolidated passage text here.",
      "source_ids": ["original_id_3", "original_id_7"],
      "has_conflict": false,
      "conflict_note": null
    }
  ],
  "merge_log": [
    {
      "merged_into": "merged_1",
      "original_ids": ["original_id_3", "original_id_7"],
      "reason": "Near-duplicate with overlapping facts about product launch date."
    }
  ],
  "token_report": {
    "original_token_count": 1240,
    "deduplicated_token_count": 890,
    "tokens_saved": 350,
    "savings_percentage": 28.2
  },
  "unmerged_passages": [
    {
      "passage_id": "original_id_2",
      "text": "Unchanged passage text."
    }
  ]
}

## CONSTRAINTS

- Do not merge passages that contain different facts, even if they share a topic.
- If two passages contradict each other, merge them but set `has_conflict` to true and add a brief `conflict_note`.
- Never invent facts not present in the original passages.
- If no duplicates exist, return all passages in `unmerged_passages` and set `deduplicated_evidence` to an empty array.
- Count tokens using the same tokenizer your pipeline uses. If unknown, estimate conservatively.

## EXAMPLES

**Example Input:**
Passage A: "The product launched on March 3, 2024."
Passage B: "Launch occurred on March 3, 2024, with 10,000 initial users."
Passage C: "The CEO announced a new pricing tier in June."

**Example Output:**
{
  "deduplicated_evidence": [
    {
      "passage_id": "merged_1",
      "text": "The product launched on March 3, 2024, with 10,000 initial users.",
      "source_ids": ["A", "B"],
      "has_conflict": false,
      "conflict_note": null
    }
  ],
  "merge_log": [
    {
      "merged_into": "merged_1",
      "original_ids": ["A", "B"],
      "reason": "Passage B contains all facts from Passage A plus additional detail."
    }
  ],
  "token_report": {
    "original_token_count": 42,
    "deduplicated_token_count": 28,
    "tokens_saved": 14,
    "savings_percentage": 33.3
  },
  "unmerged_passages": [
    {
      "passage_id": "C",
      "text": "The CEO announced a new pricing tier in June."
    }
  ]
}

To adapt this prompt for your pipeline, replace [RETRIEVED_PASSAGES] with the raw passages from your retriever, formatted as a list with unique IDs. Set [SIMILARITY_THRESHOLD] to a value like "high", "medium", or "low" depending on how aggressively you want to merge. Choose a [MERGE_STRATEGY] such as "keep_longest", "union_of_facts", or "most_recent" to control how conflicting information is resolved. Set [MAX_MERGED_TOKENS] to prevent any single merged passage from dominating the context window. If your pipeline uses a specific tokenizer, replace the token counting instruction with a callout to your own token-counting utility and remove the estimation fallback. For high-stakes domains such as healthcare or legal review, add a human-review step before the deduplicated evidence is used downstream, and consider routing passages with has_conflict: true to a separate review queue.

The evidence deduplication prompt is designed to sit between your retrieval step and your answer generation step. After you retrieve passages from your vector store, search index, or knowledge base, assemble them into the [RETRIEVED_EVIDENCE] placeholder. The prompt returns a deduplicated, merged evidence set with a token_savings estimate and merge_notes explaining what was combined. You then pass the deduplicated evidence into your downstream answer-generation prompt, reclaiming context window space for instructions, conversation history, or additional retrieval results.

Validation and retry logic: Parse the model's JSON output and validate that deduplicated_evidence is a non-empty array, each item has content and source_ids fields, and token_savings is a positive integer. If validation fails, retry once with the same input and an added [CONSTRAINT] that says 'Your previous output failed JSON schema validation. Return ONLY valid JSON matching the schema.' If the second attempt also fails, log the raw output, fall back to the original evidence set, and increment a dedup_failure metric. For high-stakes domains like healthcare or legal, route validation failures to a human review queue instead of silently falling back.

Model choice and latency: This task benefits from models with strong instruction-following and JSON output discipline. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro all perform well. For cost-sensitive pipelines, smaller models like GPT-4o-mini or Claude 3 Haiku can work if you tighten the output schema and add a retry layer. Expect 1-3 seconds of added latency. Run this prompt in parallel with your answer-generation prompt only if you can afford the extra inference cost; otherwise, run it sequentially and cache deduplication results per evidence set hash.

Logging and observability: Log the input evidence count, output evidence count, token_savings, merge notes, and validation status to your observability platform. Attach a trace ID that links the retrieval step, deduplication step, and final answer step. This lets you measure token savings over time, detect when deduplication fails to reduce evidence (indicating your retrieval step may already be diverse enough), and debug answer quality regressions by comparing answers generated with deduplicated vs. original evidence.

When to skip this prompt: If your retrieval step already returns fewer than 3 passages, or if your total context utilization is below 50% of the model's context window, the token savings may not justify the added latency and inference cost. Similarly, if your downstream task requires verbatim citation of every retrieved passage (e.g., legal document review with mandatory source listing), deduplication could obscure which specific passage supports which claim. In those cases, pass the raw evidence directly to your answer prompt and rely on citation instructions to manage attribution.