Redundant Passage Removal Prompt Template

This prompt is designed for RAG pipeline builders who need to clean a retrieved context set before it reaches the answer generation step. The core job is removing duplicate or near-duplicate passages that waste token budget and dilute the signal in the final prompt. The ideal user is an AI engineer or search operator who already has a working retrieval system (vector, keyword, or hybrid) and is observing that their top-k results contain repetitive information from overlapping document chunks, versioned documents, or semantically similar sections. You should use this prompt when your retrieval recall is acceptable but your precision and information density per token are suffering due to redundancy.

The prompt expects a list of retrieved passages as input, typically the raw output from a vector store or search index before any post-processing. It works best when passages are already segmented into coherent chunks (e.g., 200–500 tokens each) and include source metadata such as document IDs, titles, or timestamps. The output is a filtered list with removal reasons and a deduplication confidence score per removed passage, which you can log for eval and threshold tuning. Do not use this prompt as a substitute for better retrieval—if your underlying search is returning mostly irrelevant results, fix retrieval quality first with query rewriting or embedding model improvements before applying deduplication.

This prompt is not a general-purpose text summarizer or a content rewriter. It should not be used on the final answer, on user input, or on passages that have already been synthesized. It is also not appropriate when every passage in the set is known to be unique and non-overlapping—running deduplication on already-clean context wastes inference time and risks removing useful nuance. For high-stakes domains such as clinical or legal Q&A, always pair this prompt with a downstream evidence-gap check to ensure that deduplication did not remove answer-critical information. If your context set contains contradictory information from different sources, use a cross-chunk contradiction detection prompt before or after deduplication rather than relying on this prompt to resolve conflicts.

This prompt template is designed to be copied directly into your RAG pipeline. It accepts a set of retrieved passages and a user query, then returns a deduplicated, ranked list of passages with explicit removal reasons and confidence scores. The template uses square-bracket placeholders for all dynamic inputs, making it straightforward to parameterize in your application code before each model call. You should treat this prompt as a deterministic filtering step that runs after retrieval and before answer synthesis.

textCopy
You are a context deduplication engine. Your job is to remove redundant, near-duplicate, or informationally overlapping passages from a set of retrieved documents before they are used to answer a user query.

## INPUT
<query>
[QUERY]
</query>

<retrieved_passages>
[PASSAGES]
</retrieved_passages>

## CONSTRAINTS
- Preserve passages that contain unique facts, entities, or perspectives not present in other passages.
- When two passages convey the same information, keep the one that is more complete, more authoritative, or more recent.
- Do not remove a passage solely because it is short or long. Evaluate information overlap, not surface form.
- If a passage contains boilerplate, navigation text, or disclaimers with no substantive content, flag it as low-signal noise rather than a duplicate.
- Maintain source provenance for every retained passage. If you merge information from multiple sources, note this explicitly.
- Never fabricate information to fill gaps. If the deduplicated set is insufficient to answer the query, state this in the coverage assessment.

## OUTPUT SCHEMA
Return a valid JSON object with the following structure:
{
  "filtered_passages": [
    {
      "passage_id": "string (original identifier from input)",
      "source": "string (document or chunk reference)",
      "text": "string (the retained passage text)",
      "retention_reason": "string (one of: 'unique_information', 'most_complete_version', 'most_authoritative_source', 'most_recent_version', 'only_source_for_fact')",
      "deduplication_confidence": 0.0-1.0
    }
  ],
  "removed_passages": [
    {
      "passage_id": "string",
      "removal_reason": "string (one of: 'exact_duplicate', 'near_duplicate', 'information_subset', 'superseded_version', 'low_signal_noise')",
      "superseded_by_passage_id": "string or null",
      "deduplication_confidence": 0.0-1.0
    }
  ],
  "coverage_assessment": {
    "unique_facts_preserved": "integer (estimated count of distinct factual claims retained)",
    "information_loss_risk": "string (one of: 'none', 'low', 'moderate', 'high')",
    "missing_information_notes": "string or null (describe any query-relevant information that may have been lost)"
  },
  "processing_metadata": {
    "original_passage_count": "integer",
    "retained_passage_count": "integer",
    "removed_passage_count": "integer",
    "deduplication_strategy": "string (brief description of the approach used)"
  }
}

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

## INSTRUCTIONS
1. Parse all passages from the <retrieved_passages> block. Each passage must have a unique passage_id and source field.
2. Compare passages pairwise for information overlap. Two passages overlap if they convey the same factual claims, even if wording differs.
3. For each overlapping cluster, select the best passage to retain based on completeness, authority, and recency.
4. Populate the output JSON with all retained and removed passages, including reasons and confidence scores.
5. Assess whether any query-relevant information was lost during deduplication and note this in the coverage_assessment.
6. Return only the JSON object. Do not include explanatory text outside the JSON.

To adapt this template, replace each square-bracket placeholder with your application's runtime values. The [PASSAGES] placeholder expects a structured list of retrieved chunks, each with a unique identifier and source reference. The [EXAMPLES] placeholder is optional but strongly recommended for production use—include 2-3 few-shot examples showing correct deduplication decisions, especially for borderline cases like paraphrases or partial overlaps. The [RISK_LEVEL] placeholder should be set to high for regulated domains where information loss is unacceptable, which will cause the model to apply stricter retention criteria. After generating the filtered set, always validate the JSON structure before passing it to your answer generation step. If the information_loss_risk field returns high, route the original unfiltered context to a human reviewer or trigger a secondary retrieval pass with expanded queries.

This prompt is designed to sit between retrieval and answer generation in a RAG pipeline. It accepts a list of retrieved passages and returns a deduplicated set with removal reasons and confidence scores. The implementation harness must handle input assembly, output validation, retry logic, and integration with downstream answer generation. Treat this prompt as a deterministic filter: given the same input, it should produce consistent deduplication decisions. The harness should enforce this by logging removal patterns and flagging unexpected variance.

Input Assembly: The prompt expects a JSON array of passage objects, each with an id and text field. The harness should construct this from your retrieval results, preserving original document IDs and chunk indices. If your retriever returns metadata (source, page, timestamp), include it as optional fields—the prompt can use these for authority-based tiebreaking. Model Selection: Use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Smaller models may struggle with the nuanced overlap classification required. Set temperature=0 for deterministic deduplication. Output Validation: The prompt returns a JSON object with a filtered_passages array and per-passage removal_reason and confidence fields. The harness must validate: (1) all output IDs exist in the input set, (2) no duplicate IDs appear in filtered_passages, (3) confidence scores are between 0 and 1, (4) removal reasons match the allowed enum values (exact_duplicate, near_duplicate, subset_content, kept). Failed validation should trigger a retry with the validation errors included in the next prompt.

Retry and Fallback Strategy: Implement a maximum of 2 retries on validation failure. On the first retry, append the validation errors to the prompt as additional context. On the second retry, lower the confidence threshold for removal from the default (e.g., 0.7) to 0.5 to reduce false negatives. If both retries fail, fall back to a simple exact-match deduplication using text hashing and log the incident for review. Logging and Observability: Log the input passage count, output passage count, deduplication ratio, average confidence score, and any validation failures. Tag logs with the retrieval query and session ID for traceability. This data feeds into pipeline health monitoring—a sudden spike in deduplication ratio may indicate retrieval index degradation or content duplication in the source corpus.

Integration Points: The filtered output feeds directly into your answer generation prompt. Pass the filtered_passages array as the context, preserving the id fields for citation. If your answer generator expects a specific context format, add a transformation layer between this prompt's output and the answer prompt's input. Human Review Gate: For high-stakes domains (legal, clinical, financial), add an optional human review step when the deduplication ratio exceeds 0.5 (more than half of passages removed) or when any removal confidence score falls below 0.8. This catches cases where the model may have over-filtered and removed answer-critical evidence. Testing: Before production, run this harness against a golden dataset of known duplicate and unique passage pairs. Measure precision (correctly identified duplicates) and recall (all duplicates found). A recall below 0.9 indicates the prompt or model needs adjustment—consider adding few-shot examples of borderline near-duplicates to the prompt template.