Near-Duplicate Detection Prompt for Retrieved Chunks

This prompt is designed for search engineers and RAG pipeline builders who need to identify and classify semantic overlap across retrieved chunks before those chunks enter the answer-generation step. It works on a set of retrieved passages and produces a structured overlap report that classifies each pair as an exact duplicate, a paraphrase, or a partial overlap, along with a recommended retention decision. Use this when retrieval returns multiple chunks from the same document, when dense embeddings pull in near-identical passages from different sources, or when you need to reduce context-window bloat without losing unique information.

This prompt is not a general-purpose deduplication tool for entire document corpora. It assumes you have already retrieved a candidate set and need pairwise semantic comparison, not fuzzy hashing or syntactic dedup. The ideal user is a search engineer or AI engineer who has a working retrieval pipeline and is now tuning the quality of the context that enters the answer-generation step. You should have a concrete set of 5–50 retrieved passages per query, each with a unique ID and source metadata. The prompt expects you to supply these passages in a structured format, and it will return a JSON report you can use programmatically to filter, merge, or flag passages before synthesis.

Do not use this prompt when you need to deduplicate an entire document corpus at index time—that requires different techniques like document fingerprinting or embedding clustering. Do not use it when your retrieval returns fewer than three passages, as pairwise comparison adds latency without meaningful benefit. Do not use it when your passages are extremely long (over 2,000 tokens each), as the model may struggle to compare full-text semantics accurately. For those cases, consider first extracting key claims or summaries from each passage, then running this prompt on the compressed representations. The next section provides the copy-ready template you can adapt for your pipeline.

This prompt template is designed to be pasted directly into your orchestration layer. It instructs the model to analyze a set of retrieved chunks, identify pairs with high semantic overlap, and classify each pair as an exact duplicate, paraphrase, or partial overlap. The output is a structured JSON report that your application can parse to deduplicate the context window before answer generation, reducing token waste and improving answer quality by removing redundant information.

textCopy
You are a retrieval quality analyzer. Your task is to examine a set of retrieved text chunks and identify pairs that are near-duplicates. For each pair with significant semantic overlap, classify the relationship and recommend which chunk to retain.

## INPUT
[CHUNKS]

## INSTRUCTIONS
1. Compare every pair of chunks for semantic overlap. Ignore pairs where overlap is trivial (e.g., sharing only common stop words or domain boilerplate).
2. For each pair with meaningful overlap, classify the relationship as one of:
   - `exact_duplicate`: The chunks convey identical information with only minor wording differences.
   - `paraphrase`: The chunks convey the same core information but with different phrasing, structure, or level of detail.
   - `partial_overlap`: The chunks share some information but each also contains unique content not present in the other.
3. For each classified pair, recommend which chunk to retain based on:
   - Information completeness (prefer the chunk with more unique detail).
   - Authority (prefer chunks with clearer provenance or more recent dates if available).
   - Clarity (prefer well-formed, self-contained chunks over fragmented ones).
4. If two chunks are exact duplicates, retain the first occurrence by position.
5. Do not flag pairs where the overlap is below the threshold defined in [OVERLAP_THRESHOLD].

## CONSTRAINTS
- Do not modify chunk content. Only analyze and classify.
- If no pairs meet the overlap threshold, return an empty overlaps array.
- Preserve the original chunk IDs exactly as provided in the input.
- If [INCLUDE_CHUNK_TEXT] is false, reference chunks by ID only in the output.

## OUTPUT SCHEMA
Return valid JSON matching this schema:
{
  "overlaps": [
    {
      "chunk_a_id": "string",
      "chunk_b_id": "string",
      "overlap_type": "exact_duplicate | paraphrase | partial_overlap",
      "overlap_summary": "Brief description of what information the chunks share.",
      "unique_to_a": "What chunk A contains that B does not, or null if exact_duplicate.",
      "unique_to_b": "What chunk B contains that A does not, or null if exact_duplicate.",
      "retention_decision": "keep_a | keep_b | keep_both",
      "retention_reason": "One-sentence justification for the retention decision."
    }
  ],
  "total_chunks_analyzed": number,
  "total_overlap_pairs_found": number,
  "recommended_retain_count": number
}

To adapt this prompt, replace [CHUNKS] with your retrieved passages formatted as an array of objects, each containing an id and text field. Set [OVERLAP_THRESHOLD] to a value like "partial_overlap" if you want to catch all levels of redundancy, or "exact_duplicate" if you only want to remove identical content. The [INCLUDE_CHUNK_TEXT] boolean controls whether the output repeats chunk text—set to false in production to keep the response compact. After receiving the output, your application should parse the overlaps array, apply the retention_decision for each pair, and assemble a deduplicated context set using only the retained chunks. Always validate the output against the schema before using it to prune context, and log any schema violations for prompt tuning.

This prompt is designed to be called as a pre-processing step inside your retrieval pipeline, after chunks are returned from your vector store or search index but before they are assembled into the final context window for answer generation. The typical integration point is a deduplicate_context function that accepts a list of retrieved chunks and returns a filtered, deduplicated list with overlap annotations. Because the prompt operates on pairs of chunks, you will need to implement a pairing strategy—either all-pairs (O(n²)) for small retrieval sets (k ≤ 20) or a sliding-window approach that only compares chunks with high embedding cosine similarity (≥ 0.85) for larger sets. The prompt expects two chunks at a time, so your harness should iterate over candidate pairs, collect the structured overlap reports, and then apply the retention decisions to produce the final deduplicated context set.

Validation and structured output handling are critical here because downstream answer generation depends on the deduplication decisions. Parse the model's JSON output against a strict schema that checks for required fields: chunk_a_id, chunk_b_id, overlap_type (must be one of exact_duplicate, paraphrase, partial_overlap, no_overlap), overlap_score (0.0–1.0), overlapping_spans (array of text excerpts), and retention_decision (must be one of keep_both, keep_a, keep_b, merge). If the model returns invalid JSON, missing fields, or an unrecognized enum value, retry once with the validation error injected into the prompt as a [PREVIOUS_ERROR] constraint. After two failures, log the pair for human review and default to keep_both to avoid silent evidence loss. For high-stakes domains like legal or clinical RAG, route all merge decisions to a human review queue before final context assembly.

Model choice matters. This is a classification and structured extraction task that benefits from models with strong instruction-following and JSON output reliability. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro all perform well, but smaller models (e.g., GPT-4o-mini, Claude Haiku) may struggle with the overlapping_spans extraction and produce hallucinated span text. If you are using a smaller model for cost reasons, add a post-processing step that verifies each returned span actually appears in the source chunks using exact substring matching, and discard any spans that fail this check. For latency-sensitive pipelines processing many chunk pairs, consider batching up to 5 pairs in a single prompt call with a numbered list format, but be aware that batching increases the risk of cross-pair confusion—add explicit pair delimiters and validate that the output contains exactly one report per input pair.

Logging and observability should capture: the number of chunk pairs evaluated, the distribution of overlap types, the number of chunks retained vs. removed, and any validation failures or retries. Emit these metrics to your existing monitoring stack so you can track deduplication behavior over time and detect drift—for example, if exact_duplicate rates suddenly spike, your retrieval pipeline may have introduced a chunking bug. Store the full overlap reports alongside the final deduplicated context in your trace store so that downstream answer generation failures can be debugged by replaying the exact context that was assembled. When a user reports a missing fact in an answer, you can trace back through the deduplication decisions to see if critical evidence was incorrectly classified as a duplicate and removed.

What to avoid: Do not run this prompt on every retrieval call if your chunking strategy and document collection are static—cache deduplication decisions per document pair and only re-evaluate when chunks change. Do not use the merge retention decision unless you have a separate synthesis step that can faithfully combine overlapping chunks without introducing hallucinated content; in most cases, keep_a (retain the more complete or authoritative chunk) is safer. Finally, do not skip the span verification step if you are using a model that occasionally hallucinates—a single fabricated overlapping span can cause you to incorrectly merge or discard chunks, silently degrading answer quality in ways that are hard to detect without explicit eval gates.