Passage Deduplication and Redundancy Removal Prompt

This prompt is designed for RAG pipeline operators and search engineers who need to clean a retrieved passage set before it enters the context window for answer generation. The job-to-be-done is straightforward: take a list of passages returned from vector, keyword, or hybrid search and produce a deduplicated set where near-duplicate content is merged rather than repeated. Redundant context wastes token budget, dilutes signal, and often causes models to over-index on repeated information, leading to verbose or skewed answers. The ideal user is someone integrating this step into an automated retrieval pipeline—typically an engineer who already has a working retrieval system and is now hardening it for production cost and quality.

This prompt is most effective when your retrieval system returns multiple chunks from the same source document or semantically overlapping passages from different documents. It expects a structured input: a list of passages, each with a unique ID and text content. The output is a deduplicated set with merge notes explaining which passages were combined and why. Use this prompt when you observe repeated information in your context windows, when your token costs are inflated by redundant passages, or when answer quality degrades due to the model fixating on duplicated evidence. Do not use this prompt as a substitute for better retrieval. If your retriever is returning fundamentally irrelevant passages, deduplication won't fix that—you need a relevance filter or reranker upstream. Similarly, if your passages are already highly diverse and non-overlapping, this step adds latency without benefit.

Before wiring this into production, you must define what counts as a duplicate for your use case. Semantic near-duplicates (same fact, different wording) require a higher similarity threshold than exact string matches. The prompt includes a [SIMILARITY_THRESHOLD] placeholder that lets you tune this boundary. Start with a moderate threshold and adjust based on false-merge and missed-duplicate rates measured against a labeled evaluation set. Always run the deduplication step before context assembly, not after, and log merge decisions for debugging retrieval quality over time. If your domain includes passages where near-duplicate content carries different metadata, authority levels, or timestamps, you may need to extend the prompt with merge rules that preserve the most authoritative or recent version rather than simply combining text.

This section provides a copy-ready prompt template you can adapt directly into your retrieval pipeline. The prompt is designed to take a batch of retrieved passages, identify exact duplicates and near-duplicates, and produce a cleaned set with merge notes. It uses square-bracket placeholders for all variable inputs so you can wire it into your application without rewriting the core instruction logic.

textCopy
You are a passage deduplication engine for a RAG pipeline. Your job is to clean a set of retrieved passages before they are used for answer generation.

## INPUT
You will receive a list of passages. Each passage has an ID and text content.

[PASSAGES]

## TASK
1. Identify exact duplicate passages (identical or near-identical text).
2. Identify near-duplicate passages that convey the same information with minor wording differences.
3. For each group of duplicates or near-duplicates, select the single best passage to keep based on:
   - Completeness of information
   - Clarity of expression
   - Absence of contradictory statements within the passage
4. For near-duplicates, produce a brief merge note explaining what additional information the discarded passages contained, if any.
5. Return all unique, non-duplicate passages unchanged.

## CONSTRAINTS
- Do not merge passages that convey different facts, even if they share a topic.
- Do not discard a passage unless you are confident it is a duplicate or near-duplicate.
- Preserve the original passage text exactly for kept passages. Do not rewrite or summarize.
- If uncertain whether two passages are duplicates, keep both and flag them for human review.

## OUTPUT FORMAT
Return a JSON object with the following structure:

{
  "kept_passages": [
    {
      "id": "original passage ID",
      "text": "original passage text unchanged",
      "status": "kept | merged",
      "merged_from_ids": ["list of discarded passage IDs"],
      "merge_note": "explanation of what was merged, or null if kept as-is"
    }
  ],
  "discarded_passages": [
    {
      "id": "discarded passage ID",
      "reason": "exact_duplicate | near_duplicate",
      "merged_into_id": "ID of the kept passage"
    }
  ],
  "flagged_for_review": [
    {
      "id": "passage ID",
      "reason": "why this passage could not be confidently classified"
    }
  ]
}

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

To adapt this template, replace [PASSAGES] with your serialized passage list, typically as a JSON array of objects with id and text fields. The [EXAMPLES] placeholder should contain 2-3 few-shot examples showing correct deduplication behavior, including a borderline near-duplicate case. Set [RISK_LEVEL] to high if downstream answers have regulatory or safety implications—this should trigger additional human review flags in your harness. For production use, validate the output JSON against the schema before passing deduplicated passages to your answer generation step. If the model returns malformed JSON, use a repair prompt or retry with stricter format instructions rather than silently falling back to the raw retrieval set.

The deduplication prompt should sit between retrieval and answer generation in your RAG pipeline. After your retriever returns a set of passages, batch them into groups that fit within your model's context window—typically 10–30 passages per call depending on passage length. The prompt expects a list of passages with IDs, so assign a stable identifier (e.g., doc_7_chunk_3) to each passage before calling the prompt. This ID is critical for traceability: downstream components need to know which original passages were merged or retained.

Wrap the prompt call in a validation layer that checks the output schema before passing results to the answer generator. The prompt returns a JSON object with kept_passages and merge_notes. Validate that every kept_passage_id references an ID that existed in the input, that no passage ID appears in both kept_passages and merge_notes.merged_ids, and that every merged_id in merge_notes also existed in the input. If validation fails, retry once with the same input and a stronger constraint instruction appended to the prompt. If the second attempt also fails, log the failure and fall back to using the raw, undeduplicated passage set with a warning flag.

For model choice, use a model with strong instruction-following and JSON output capabilities. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro all perform well on this structured deduplication task. Avoid smaller models (under 7B parameters) for production use unless you've calibrated them extensively—they tend to over-merge dissimilar passages or miss near-duplicates with minor wording differences. If cost is a concern, consider using a cheaper model for the initial deduplication pass and a stronger model as a judge to spot-check 5–10% of outputs for false merges and missed duplicates.

Logging and observability are essential. For every deduplication call, log: the input passage count, the output passage count, the merge decisions (which IDs were merged and why), the model used, latency, and any validation failures. This data lets you monitor deduplication ratios over time and detect drift—for example, if your retriever starts returning more near-duplicates from a new data source, you'll see the merge rate spike. Set up an alert if the deduplication ratio (output count / input count) drops below 0.3 or exceeds 0.95, as both extremes suggest the prompt may be misbehaving.

For evaluation, maintain a golden dataset of passage sets with known duplicates. Run the prompt against this dataset weekly or on every prompt change. Measure precision (did we avoid merging passages that aren't actually duplicates?) and recall (did we catch all the duplicates we should have?). A false merge—combining two distinct passages—is more harmful than a missed duplicate, so weight precision higher in your acceptance criteria. If your application is in a regulated domain, route all merge decisions to a human review queue for the first month of production to calibrate trust before allowing automated deduplication to feed directly into answer generation.