Duplicate Context Chunk Detection Prompt for RAG

This prompt is for RAG pipeline engineers who need to detect and handle duplicate or near-duplicate content chunks before they are injected into a model's context window. The core job-to-be-done is preventing context window waste and repetitive, low-information answers caused by a retrieval system returning highly similar text segments from a vector database or search index. The ideal user is a backend engineer or MLOps specialist who controls the retrieval-augmented generation assembly step and can insert a deduplication check between the retriever and the LLM call.

Use this prompt when you have a list of retrieved context chunks—each with chunk_id, text, and optional source_metadata—and you suspect your retrieval pipeline is pulling overlapping passages from the same document or semantically identical information from different sources. The prompt is designed to be stateless and idempotent: you pass in a batch of chunks, and it returns a deduplicated subset with an explanation of which chunks were removed and why. It is not designed for deduplicating final model outputs, user-generated content, or database records. For those tasks, refer to sibling playbooks like Exact Duplicate Removal or Near-Duplicate Paragraph Detection.

Do not use this prompt when the duplication is intentional, such as when multiple sources corroborate a critical fact and you want the model to see the consensus. The prompt's overlap tolerance is configurable via the [OVERLAP_THRESHOLD] parameter, but it will aggressively collapse near-duplicates by default. Also, avoid this prompt if your chunks are already small and the deduplication overhead outweighs the context savings. For high-throughput systems, consider a pre-filter using MinHash or embedding cosine similarity before invoking an LLM for the final semantic judgment. The next section provides the copy-ready template you can adapt and integrate directly into your RAG assembly harness.

This template is designed to be dropped directly into a RAG pipeline's context assembly stage. It instructs the model to act as a duplicate detection filter over a set of retrieved chunks, identifying pairs or groups that convey substantially the same information. The prompt is structured to return a machine-readable JSON output, making it suitable for automated pre-processing before the final LLM call. The core logic separates exact duplicates from near-duplicates, preserving source attribution so downstream merging logic can decide which chunk to keep without losing provenance.

textCopy
You are a context deduplication filter for a RAG (Retrieval-Augmented Generation) system. Your task is to analyze a set of retrieved context chunks and identify any that contain duplicate or near-duplicate information.

## INPUT
You will receive a JSON array of context chunks. Each chunk has an `id`, `source`, and `content`.

[CHUNKS]

## TASK
1. Compare all chunks pairwise to detect semantic duplication.
2. Classify each duplicate pair as `exact` (verbatim or trivial rewording) or `near_duplicate` (same core facts, different phrasing).
3. For each duplicate group, identify which chunk to keep based on [RETENTION_RULE] (e.g., `longest`, `highest_confidence`, `preferred_source`).
4. Do not flag chunks as duplicates if they share a topic but contain different, non-overlapping facts.

## OVERLAP TOLERANCE
Consider two chunks near-duplicates if their semantic overlap exceeds [OVERLAP_THRESHOLD]%.

## OUTPUT_SCHEMA
Return a valid JSON object with this exact structure:
{
  "duplicate_groups": [
    {
      "group_id": "string",
      "type": "exact | near_duplicate",
      "chunk_ids": ["string"],
      "keep_chunk_id": "string",
      "rationale": "string"
    }
  ],
  "unique_chunks": ["string"]
}

## CONSTRAINTS
- Only output the JSON object. No markdown, no extra text.
- The `unique_chunks` array must list every input chunk `id` that is not part of any duplicate group.
- Every input chunk `id` must appear exactly once across `duplicate_groups` and `unique_chunks`.

To adapt this template, replace the [CHUNKS] placeholder with your serialized chunk array. The [RETENTION_RULE] placeholder should be set to a deterministic policy like keep_longest or keep_highest_confidence; avoid ambiguous instructions that could cause non-deterministic selection. The [OVERLAP_THRESHOLD] is a tunable integer between 0 and 100—start at 80 for near-duplicate detection and adjust based on your domain's tolerance for redundancy. For high-stakes applications, add a confidence field to the output schema and route groups with low confidence to a human review queue. Always validate the output JSON against the schema before proceeding to context assembly; a malformed response here can silently pass duplicates into the final prompt.

This prompt is designed to sit between your retrieval step and your answer generation step. It should be called after you have retrieved a set of candidate chunks from your vector store or search index but before you assemble the final prompt for your generation model. The core job is to prevent wasted context window space and reduce repetitive answers by identifying and flagging chunks that are duplicates or near-duplicates. The prompt expects a list of chunks, each with a unique chunk_id, the text content, and optional source_attribution metadata. It returns a structured JSON payload that includes a keep list, a duplicate_pairs list, and a deduplication_summary.

To wire this into a production application, wrap the LLM call in a function that accepts a list of chunk objects and returns a deduplicated list. The function should first validate the input: ensure every chunk has a non-empty chunk_id and text field. If the total token count of the input chunks is below a configurable threshold (e.g., 2,000 tokens), you can skip the deduplication call entirely to save latency and cost. For the model choice, a fast and inexpensive model like GPT-4o-mini or Claude 3.5 Haiku is usually sufficient for this classification task. Implement a strict JSON schema validation on the output. If the model response fails to parse or the keep list references chunk_id values not present in the input, log the failure and fall back to using the original, un-deduplicated chunk list to avoid breaking the generation pipeline. The overlap_tolerance parameter in the prompt should be mapped to a configuration variable in your application, allowing operators to tune sensitivity without changing the prompt text.

For observability, log the deduplication_summary metrics—specifically original_chunk_count, kept_chunk_count, and duplicate_pairs_count—as structured log attributes or metrics. This allows you to monitor the deduplication rate over time and detect if your retrieval system is producing an increasing number of redundant chunks. If the deduplication rate spikes suddenly, it may indicate an indexing problem or a shift in your document corpus. In high-stakes applications where source attribution is critical, add a post-processing step that verifies no source_attribution value was lost during deduplication. If a chunk is removed as a duplicate, ensure its source is still represented by the kept chunk. If not, flag the pair for human review or adjust the prompt's overlap_tolerance downward to be more conservative.