Context Deduplication Harness Prompt for RAG Pipelines

This prompt is for AI architects and RAG pipeline builders who need a single, composable system prompt to clean a set of retrieved passages before they reach the answer generation step. It orchestrates three filtering operations in sequence: redundancy removal, noise filtering, and uniqueness ranking. Use it when your retrieval system returns overlapping chunks, boilerplate text, or low-signal content that degrades answer quality. This harness is designed to be placed between retrieval and synthesis in a production pipeline, accepting a raw context list and returning a structured, deduplicated context payload ready for downstream consumption.

This is not a replacement for embedding-based retrieval tuning or a vector database re-ranking module; it is a prompt-level quality gate for context hygiene. It works best when your retrieval system already returns a candidate set that is broadly on-topic but suffers from chunk overlap, repeated information across document versions, or the inclusion of navigational boilerplate. The harness expects a list of passages with source metadata and a user query as input. It returns a JSON payload containing the filtered passages, removal reasons for each excluded item, and a uniqueness coverage report. You should wire this prompt into your pipeline with a strict output schema validator and a token budget check before passing the cleaned context to your answer generation model.

Do not use this prompt when your retrieval system returns fewer than five passages, when passages are already highly distinct, or when your downstream answer model has a context window large enough to absorb redundancy without quality loss. It is also inappropriate for real-time streaming applications where the added latency of an LLM call for deduplication would violate latency budgets. For those cases, consider a deterministic deduplication method like exact substring matching or embedding-based cosine similarity thresholds. If your use case involves regulated content where removal decisions must be auditable, pair this prompt with the Context Provenance Tracking Prompt After Deduplication to maintain a full attribution chain. Start by running this harness on a golden dataset of retrieval results and measuring the impact on downstream answer accuracy and faithfulness before deploying to production.

This prompt template is designed to sit between your retrieval step and your answer generation step in a RAG pipeline. It accepts a set of retrieved passages and a user query, then produces a cleaned, deduplicated, and ranked context set ready for downstream synthesis. The prompt is structured to produce machine-readable JSON output so your application can validate the result before passing it to the answer generator. Replace every square-bracket placeholder with your actual inputs, output schema, and operational constraints before deploying.

textCopy
You are a context-cleaning engine in a retrieval-augmented generation pipeline. Your job is to process a set of retrieved passages and produce a filtered, deduplicated, and ranked context set for downstream answer generation.

## INPUT
- User Query: [QUERY]
- Retrieved Passages (with IDs): [RETRIEVED_PASSAGES]
- Maximum Token Budget for Output Context: [MAX_OUTPUT_TOKENS]
- Minimum Relevance Threshold (0.0–1.0): [MIN_RELEVANCE_SCORE]

## CONSTRAINTS
[CONSTRAINTS]

## OUTPUT SCHEMA
Return a valid JSON object with this structure:
{
  "filtered_passages": [
    {
      "passage_id": "string",
      "text": "string",
      "relevance_score": 0.0,
      "uniqueness_score": 0.0,
      "retention_reason": "unique_information | highest_authority | most_recent | only_source_for_fact",
      "source_document": "string"
    }
  ],
  "removed_passages": [
    {
      "passage_id": "string",
      "removal_reason": "duplicate | near_duplicate | below_relevance_threshold | noise | distractor | outdated_version",
      "duplicate_of_passage_id": "string | null"
    }
  ],
  "coverage_assessment": {
    "query_aspects_covered": ["string"],
    "query_aspects_missing": ["string"],
    "overall_context_quality": "sufficient | partial | insufficient"
  },
  "token_count": 0
}

## INSTRUCTIONS
1. Score each passage for relevance to the query on a 0.0–1.0 scale. Remove passages below [MIN_RELEVANCE_SCORE].
2. Identify duplicate or near-duplicate passages. For each cluster of overlapping content, retain only one passage based on these priority rules: [DEDUPLICATION_PRIORITY_RULES].
3. Classify removed passages with specific removal reasons.
4. For each retained passage, calculate a uniqueness score indicating how much new information it contributes beyond other retained passages.
5. Rank retained passages by relevance, then by uniqueness within relevance tiers.
6. Assemble the final context set, staying within [MAX_OUTPUT_TOKENS]. If the token budget forces cuts, drop the lowest-ranked passages first and note any lost coverage in the coverage_assessment.
7. Flag any query aspects that no retained passage addresses.
8. If the overall context quality is "insufficient," do not fabricate passages. Report the gap.

## EXAMPLES
[EXAMPLES]

Return only the JSON object. No preamble, no commentary.

Adaptation notes: The [CONSTRAINTS] placeholder should include domain-specific rules such as 'prefer peer-reviewed sources over blog posts' or 'retain regulatory filings over news summaries.' The [DEDUPLICATION_PRIORITY_RULES] field is where you encode your source authority hierarchy—for example, 'official documentation > internal wiki > external articles.' The [EXAMPLES] placeholder should contain one or two worked examples showing the expected output shape for your domain. If your pipeline cannot tolerate any evidence loss, set [MIN_RELEVANCE_SCORE] low and rely on the coverage_assessment to surface gaps rather than aggressively filtering. Always validate the output JSON against your schema before forwarding to answer generation; a malformed deduplication output will break downstream synthesis.

The Context Deduplication Harness is designed to sit between retrieval and answer generation in your RAG pipeline. It accepts a list of retrieved passages and returns a cleaned, deduplicated, and ranked context set. The harness is not a standalone prompt—it is a composable system prompt that orchestrates multiple filtering steps in a single pass. You should call it after retrieval and before answer synthesis, passing the raw retrieval results along with the user query and any metadata your pipeline tracks (source IDs, retrieval scores, timestamps). The output is a structured JSON payload containing the filtered passages, removal reasons, uniqueness scores, and a quality gate decision that downstream components can act on.

Pipeline Integration: Wire the harness as a synchronous step in your context assembly function. In pseudocode: raw_contexts = retriever.search(query, top_k=20); filtered = dedup_harness(query, raw_contexts, token_budget=4000); answer = generator(query, filtered.passages). The harness expects a list of passage objects with at minimum id, text, and source fields. You can optionally pass retrieval_score and timestamp for temporal deduplication. The prompt template uses placeholders like [QUERY], [PASSAGES], and [TOKEN_BUDGET]—populate these from your application layer before sending the request. Choose a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). For cost-sensitive pipelines, consider using a smaller model for the initial noise classification step and the full harness only when the noise ratio exceeds a threshold.

Validation and Error Handling: The harness outputs a JSON schema with filtered_passages, removal_log, quality_gate, and coverage_report. Validate this output before passing it to answer generation. Check that quality_gate.passed is true—if false, inspect quality_gate.failure_reasons and decide whether to expand retrieval, adjust thresholds, or escalate. Implement a retry strategy: if the model returns malformed JSON, retry once with a repair prompt that includes the raw output and schema. If the harness removes more than 50% of passages, trigger an over-filtering check using the sibling Over-Filtering Risk Assessment Prompt to verify that answer-critical evidence wasn't stripped. Log every harness invocation with the input passage count, output passage count, quality gate decision, and latency for monitoring.

Production Hardening: Add a circuit breaker: if the harness fails three consecutive times or exceeds a latency threshold (e.g., 5 seconds), fall back to a simpler top-k-by-score selection and flag the incident. Cache harness results when the same query and retrieval set appear within a short window—this is common in conversational RAG where users ask follow-ups against the same retrieved context. For high-throughput systems, batch multiple harness calls where queries are independent. Always include source provenance in the output so downstream answer generation can cite original documents even after deduplication merges or removes passages. The harness's removal_log should be stored alongside the final answer for auditability—it explains exactly why each passage was kept or dropped.

What to avoid: Don't use the harness on retrieval sets smaller than 5 passages—the overhead isn't justified and the deduplication signal is too weak. Don't skip the quality gate check before answer generation; a failed gate means the context is unreliable. Don't use the harness as a replacement for improving your retriever—if noise ratios consistently exceed 40%, invest in better retrieval, not just better filtering. Finally, don't treat the harness output as the final answer context without human review in high-stakes domains (healthcare, legal, finance). The harness reduces noise but doesn't guarantee completeness—always pair it with an evidence gap check before presenting answers to users in regulated workflows.