Conversation History Deduplication Prompt for RAG

This prompt is designed for RAG pipeline builders who observe redundant retrieval across conversation turns. The core job-to-be-done is filtering the current turn's retrieved passages to remove any evidence that has already been cited in prior answers, producing a deduplicated evidence set. The ideal user is an AI engineer or backend developer who controls the retrieval-to-generation handoff and has access to both the conversation history and the current retrieval results. Required context includes: the full conversation history with explicit source citations in prior assistant messages, the current user query, and the raw list of retrieved passages for the current turn. Without citation tracking in your history, this prompt cannot function because it relies on comparing new passage identifiers or content against previously cited sources.

Deploy this prompt when your RAG system exhibits answer repetition across turns, when context windows are tight and you need to maximize novel information density, or when you need to guarantee that each turn's answer adds incremental value beyond what was already stated. The prompt acts as a filter between retrieval and generation—it should be inserted after retrieval but before the generator prompt assembles its final context. For implementation, you will need to pass the conversation history as a structured list of prior assistant messages with their citation blocks, and the current turn's retrieved passages with unique identifiers (such as chunk IDs, document IDs, or content hashes). The prompt compares these and outputs only the passages that contain information not already covered by previously cited sources. You should validate the output by checking that no passage ID in the deduplicated set appears in the prior citations list, and that the total token count of the deduplicated set is less than or equal to the original retrieval set.

Do not use this prompt for single-turn Q&A systems where there is no conversation history to deduplicate against. Do not use it when your RAG pipeline lacks citation tracking or when prior answers do not contain explicit source references. Avoid this prompt when the conversation history is extremely long and the cost of processing all prior citations exceeds the benefit of deduplication—in those cases, consider a session summarization prompt first. Also avoid this prompt when retrieved passages contain time-sensitive information where even identical sources may need re-citation due to temporal context changes. The next step after implementing this prompt is to wire it into your RAG pipeline's context assembly stage and add eval checks for novelty detection and information loss prevention, as described in the implementation harness section.

This prompt template sits between your retrieval step and answer generation in a multi-turn RAG pipeline. Its job is to compare a freshly retrieved set of evidence chunks against the evidence already cited in previous conversation turns, then output a deduplicated set containing only novel passages. Use this when your retriever returns overlapping or identical chunks across turns, causing repetitive answers that waste context budget and erode user trust.

textCopy
You are a retrieval deduplication filter for a multi-turn RAG system. Your task is to compare a set of newly retrieved evidence passages against passages already cited in prior conversation turns. Output only the novel passages that contain information not already covered by the previously cited evidence.

[CONVERSATION_HISTORY]

[RETRIEVED_EVIDENCE]

[OUTPUT_SCHEMA]

[CONSTRAINTS]

[EXAMPLES]

Adapt this template by populating the placeholders with your specific pipeline data. [CONVERSATION_HISTORY] should contain the prior turns with their cited source IDs and key claims. [RETRIEVED_EVIDENCE] should be the raw retrieval results for the current turn, each with a unique passage ID. [OUTPUT_SCHEMA] must define the exact JSON structure you expect, including fields for passage ID, content, and a novelty flag. [CONSTRAINTS] should specify your deduplication rules: whether near-duplicates count as duplicates, how to handle partial overlap, and whether to preserve or discard passages that add minor new details. [EXAMPLES] should include at least one case where a passage is kept and one where it is discarded, showing the reasoning. After pasting this into your pipeline, validate the output against your schema before passing it to the answer generation step. For high-stakes domains, add a human review gate when the deduplication rate exceeds 80%, as aggressive filtering may indicate information loss.

The deduplication prompt should sit between your retrieval step and your answer generation step, operating as a filter that reduces the evidence payload before it reaches the synthesis model. In a typical RAG pipeline, you retrieve chunks for each user turn, but many of those chunks will overlap with passages already cited in prior answers. Running this prompt before answer generation prevents the model from re-synthesizing stale evidence and keeps your context window focused on genuinely new information. The prompt expects a structured input containing the current query, the newly retrieved passages, and a summary of previously cited evidence from the conversation history. You should assemble this input programmatically from your session store and retrieval index rather than asking the user to provide it.

Wire the prompt into your pipeline as a synchronous pre-processing step with a strict output contract. The prompt returns a JSON object with a deduplicated_passages array and a removed_passages array, each containing passage IDs and a brief reason for inclusion or removal. Implement a JSON schema validator immediately after the model response to enforce this structure. If validation fails, retry once with the validation error message appended to the prompt as a correction hint. Log every deduplication decision—including which passages were removed and why—to your observability platform so you can audit for information loss. For high-stakes domains where removing a passage could drop critical evidence, add a secondary check: if the deduplication step removes more than 50% of retrieved passages, flag the turn for human review or re-retrieval with adjusted similarity thresholds before proceeding to answer generation.

Model choice matters here. This is a classification-and-filtering task that benefits from deterministic behavior, so prefer models with strong instruction-following and low temperature settings (0.0–0.2). Avoid models that tend to over-explain or add commentary outside the requested JSON schema. If you're using a model that doesn't reliably produce valid JSON, wrap the call in a retry loop with schema validation and consider using structured output APIs like function calling or constrained decoding to enforce the output shape. Do not let a deduplication failure silently pass through to the answer generation step—a failure here means your synthesis model will receive redundant context, which defeats the purpose of the prompt and wastes token budget on already-cited evidence. Build a circuit breaker: if deduplication fails after two retries, log the failure, pass the full retrieval set to answer generation with a warning flag, and alert the on-call channel so the pipeline degradation is visible.