Evidence Deduplication and Ranking Prompt

This prompt is designed for RAG pipelines that retrieve from overlapping document collections, mirrored content, or versioned knowledge bases where near-duplicate passages inflate retrieval sets. When multiple retrieved passages convey the same information, downstream answer generation can over-weight that evidence, producing skewed or repetitive answers. This prompt identifies semantically redundant passages, selects the strongest representative from each duplicate cluster, and ranks the resulting unique evidence by relevance, specificity, and support strength. Use this prompt after retrieval and before answer generation. It assumes you already have a retrieval set with passage IDs and source metadata.

Do not use this prompt when every passage is known to be unique or when deduplication is handled at the retrieval index level. If your vector store already applies distinct filters, exact-match hashing, or embedding-based deduplication at query time, running this prompt adds latency without benefit. Similarly, skip this prompt for single-source retrieval sets where passage overlap is structurally impossible. The prompt is most valuable when your retrieval pipeline pulls from multiple indexes, document versions, or mirrored corpora where the same fact appears in several nearly identical forms. In those cases, deduplication before ranking prevents a single piece of evidence from dominating the answer simply because it was stored in three places.

Before wiring this into production, verify that your retrieval set includes passage IDs and source metadata. The prompt expects structured input with identifiable passages. If your retrieval layer returns raw text without identifiers, add a pre-processing step to assign stable IDs. Also plan for the token cost: deduplication requires the model to compare all passages against each other, so very large retrieval sets (50+ passages) may need chunking or a pre-filter step. Start with retrieval sets of 10–30 passages and measure whether deduplication improves answer quality before scaling up. If answer quality doesn't change, your retrieval set likely has low redundancy and you can remove this step.

This prompt template is designed to be pasted directly into your system instructions or user message for an LLM call. It instructs the model to identify near-duplicate or semantically redundant passages from a retrieval set, select the strongest representative from each cluster, and produce a final ranked list of unique evidence. The template uses square-bracket placeholders that you must replace with your specific data before sending the request to the model.

textCopy
You are an evidence deduplication and ranking engine for a retrieval-augmented generation (RAG) system. Your task is to process a set of retrieved passages, identify near-duplicates or semantically redundant entries, select the strongest representative from each cluster, and rank the unique evidence by relevance, specificity, and support strength for the given query.

## INPUT

**Query:** [USER_QUERY]

**Retrieved Passages:**
[PASSAGE_LIST]
- Each passage must include a unique ID and the full text.
- Format: `[PASSAGE_ID]: [PASSAGE_TEXT]`

## TASK STEPS

1. **Semantic Comparison:** Compare all passages pairwise. Identify clusters of passages that convey substantially the same information, even if wording differs. Two passages are duplicates if they would support the same claim with no meaningful additional detail.

2. **Representative Selection:** For each duplicate cluster, select the single strongest representative based on:
   - **Specificity:** Prefer passages with concrete details, numbers, dates, or named entities over vague statements.
   - **Completeness:** Prefer passages that contain the full claim rather than partial fragments.
   - **Source Quality:** If source metadata is provided, prefer higher-authority sources.
   - **Query Alignment:** Prefer passages that more directly address the query.

3. **Uniqueness Verification:** Confirm that all remaining passages after deduplication are semantically distinct. If two passages still overlap significantly, merge them into the same cluster.

4. **Relevance Ranking:** Rank the unique passages by their relevance and support strength for the query. Assign each passage a rank position starting from 1 (most relevant).

## OUTPUT SCHEMA

Return a JSON object with the following structure:

```json
{
  "query": "[USER_QUERY]",
  "total_input_passages": [NUMBER],
  "duplicate_clusters_found": [NUMBER],
  "unique_passages_count": [NUMBER],
  "ranked_evidence": [
    {
      "rank": 1,
      "passage_id": "[PASSAGE_ID]",
      "passage_text": "[FULL_PASSAGE_TEXT]",
      "relevance_score": [0.0-1.0],
      "strength_tier": "[high|medium|low]",
      "deduplication_note": "[original|selected_representative_of_cluster_X]",
      "rationale": "[One sentence explaining why this passage is ranked here.]"
    }
  ],
  "removed_duplicates": [
    {
      "removed_passage_id": "[PASSAGE_ID]",
      "replaced_by_passage_id": "[PASSAGE_ID]",
      "reason": "[Why this passage was removed as a duplicate.]"
    }
  ]
}

CONSTRAINTS

• Do not invent or modify passage text. Use the exact text provided.
• If no duplicates are found, set duplicate_clusters_found to 0 and include all passages in the ranked list.
• If the entire retrieval set is empty, return an empty ranked_evidence array and set counts to 0.
• Assign relevance_score values that are calibrated and distinguishable. Avoid giving all passages the same score.
• The strength_tier must be one of: high, medium, low.
• Every removed duplicate must appear in the removed_duplicates array with a clear reason.
• If source metadata is provided in the passage list, use it for authority assessment. If not provided, rely only on passage content.

EXAMPLE

Query: "What are the side effects of drug X?"

Passages:

• P1: "Drug X may cause nausea and headache in some patients."
• P2: "Common side effects of Drug X include nausea, headache, and dizziness."
• P3: "Clinical trials show Drug X is effective for condition Y with mild side effects."

Expected Deduplication: P1 and P2 are near-duplicates. P2 is the stronger representative because it includes dizziness as an additional side effect and is more complete. P3 is distinct because it discusses efficacy, not side effects.

Expected Ranking: P2 (rank 1, directly addresses side effects with specific list), P3 (rank 2, mentions side effects tangentially but provides clinical context), P1 removed as duplicate.

Now process the input passages and return the JSON output.

Adaptation guidance: Replace [USER_QUERY] with the original question or claim being evaluated. Replace [PASSAGE_LIST] with your retrieved passages, ensuring each has a unique ID and the full text. If your retrieval system provides source metadata such as publication date, author, or domain authority score, include it alongside each passage and adjust the representative selection criteria accordingly. For high-stakes domains like medical or legal, add a [RISK_LEVEL] parameter and require human review when confidence is low or when the deduplication removes passages that might contain critical nuance. Always validate the output JSON against the schema before passing ranked evidence to downstream answer generation.

This prompt is designed to sit between retrieval and answer generation in a RAG pipeline. It consumes a flat list of retrieved passages and outputs a deduplicated, ranked list of unique evidence items. The implementation harness must enforce a strict contract: the prompt receives a JSON array of passages with IDs, text, and optional metadata, and it must return a JSON array of ranked, unique passages with deduplication rationale. Because this step directly shapes the context window for downstream generation, the harness must validate the output before any answer is produced.

Wire the prompt into your application as a post-retrieval processor. After your retriever returns N passages, serialize them into the [RETRIEVED_PASSAGES] placeholder as a JSON array with fields: id, text, source, and optional score or date. Call the model with a low temperature (0.0–0.2) to maximize deterministic deduplication. Parse the response and validate the JSON schema: each output item must contain id, text, rank, deduplication_decision (either kept or removed_as_duplicate_of), and rationale. If the output fails schema validation, retry once with a repair prompt that includes the validation error. If the second attempt fails, log the failure and fall back to a simpler deduplication heuristic (e.g., exact text hash or embedding cosine similarity above 0.95) to avoid blocking the pipeline.

Logging and observability are critical here because deduplication errors compound downstream. Log the input passage count, output passage count, deduplication decisions, and any validation failures. Include a trace ID that links the retrieval set, deduplication output, and final generated answer. For high-stakes domains such as legal or medical, route outputs where more than 30% of passages were flagged as duplicates to a human review queue before answer generation. Avoid running this prompt on retrieval sets larger than 20–30 passages without chunking, as model context limits and ranking stability degrade with very long input lists. If your retriever returns more passages, pre-filter by relevance score before deduplication.