Entity-Centric Deduplication Prompt for Factual Q&A

This prompt is designed for factual QA systems where retrieval returns multiple passages that mention the same entities with overlapping facts. Instead of generic semantic deduplication, this prompt clusters passages by the entities they discuss and retains only one representative passage per unique entity-fact pair. Use this when your retrieval pipeline pulls 20-50 chunks from a dense vector store and many chunks repeat the same entity-attribute combinations—for example, five passages all stating that 'Acme Corp was founded in 2004' or 'the API endpoint returns JSON.' The prompt sits between retrieval and answer generation, reducing context bloat while preserving entity coverage so the downstream answer synthesizer has access to the widest range of distinct facts rather than wasting tokens on redundant mentions.

The ideal deployment scenario is a product knowledge base, documentation site, or structured factual corpus where entities are well-defined and facts about them are atomic. For instance, a customer-facing Q&A system for a SaaS product where retrieval pulls overlapping documentation paragraphs about features, pricing, and limits. The prompt expects passages that already have reasonable relevance to the query—it assumes your retriever has done its job. You wire it in as a preprocessing step: retrieved chunks go in, a deduplicated set comes out, and that set feeds into your answer generation prompt. The output should include the retained passages plus a mapping of which entities were covered, so you can validate that no critical entity was dropped during deduplication.

Do not use this prompt when passages contain complex multi-entity relationships that require cross-passage reasoning. Entity-centric clustering can fragment relational context—if Passage A says 'Acme Corp acquired Beta Inc' and Passage B says 'Beta Inc launched Product X in 2023,' deduplicating by entity may drop one passage and break the acquisition-to-product chain. Similarly, avoid this prompt for narrative content, legal documents where clause relationships matter, or any domain where facts derive meaning from their surrounding context rather than standing alone. If your downstream task requires comparing entities or reasoning about how facts interact, use a context prioritization or maximal marginal relevance approach instead. Before deploying, run an over-filtering risk assessment: take a sample of queries, compare answer quality with and without deduplication, and check whether entity coverage improved or relational context was lost.

This prompt template is designed to be pasted into your context-cleaning step before answer generation. It takes a set of retrieved passages and a user query, then produces a deduplicated context set organized around unique entity-fact pairs. The goal is to preserve entity coverage for downstream answer completeness while eliminating redundant passages that add token cost without new information. Use this when your retrieval pipeline returns multiple chunks covering the same entities with overlapping facts—common in dense document collections, versioned knowledge bases, and multi-source RAG systems.

textCopy
You are a context deduplication engine for a factual question-answering system. Your job is to remove redundant passages from a retrieved context set while preserving unique entity-fact coverage.

INPUT:
- User Query: [QUERY]
- Retrieved Passages (each with a passage_id and text): [PASSAGES]

INSTRUCTIONS:
1. Extract all distinct entities mentioned across the passages. An entity is a named person, organization, location, product, date, event, or concept that carries factual weight for the query.
2. For each entity, identify the unique factual claims made about it across all passages. A factual claim is a verifiable statement that adds information not already captured by another passage for the same entity.
3. For each unique entity-fact pair, select exactly one passage that best represents that fact. Prefer passages that are:
   - Most specific and complete for that fact
   - From the most authoritative or recent source if source metadata is available
   - Self-contained enough to be understood without surrounding context
4. Discard all passages that contain only redundant facts already covered by a selected passage.
5. If a passage contains multiple entity-fact pairs and some are redundant while others are unique, you may retain the passage but note which facts are the unique contribution.

OUTPUT_SCHEMA:
{
  "deduplicated_passages": [
    {
      "passage_id": "string",
      "text": "string",
      "entities_covered": ["entity_name"],
      "unique_facts": ["fact summary"],
      "retention_reason": "unique_entity_fact | best_representation | only_source"
    }
  ],
  "removed_passages": [
    {
      "passage_id": "string",
      "removal_reason": "fully_redundant | no_unique_facts | superseded_by",
      "superseded_by_passage_id": "string or null"
    }
  ],
  "entity_coverage_summary": {
    "total_entities_found": 0,
    "entities_with_coverage": ["entity_name"],
    "entities_missing_coverage": ["entity_name or empty"]
  },
  "deduplication_stats": {
    "original_passage_count": 0,
    "retained_passage_count": 0,
    "removed_passage_count": 0,
    "entity_fact_pairs_retained": 0
  }
}

CONSTRAINTS:
- Never discard a passage if it is the only source for an entity relevant to the query.
- If two passages contain the same fact but one adds qualifying detail, retain the more detailed one.
- Do not merge or rewrite passage text. Return the original text of retained passages.
- If the query asks about multiple entities, ensure each queried entity has at least one retained passage if available.
- If source date or authority metadata is provided in the passage, use it to break ties.

To adapt this template, replace [QUERY] with the user's original question and [PASSAGES] with your retrieved context set, ensuring each passage has a unique passage_id and its full text. If your retrieval system provides source metadata such as document dates, authority scores, or source names, include those fields in the passage objects and update the tie-breaking criteria in the instructions accordingly. For high-stakes domains such as legal, clinical, or financial Q&A, add a human review step before the deduplicated context is passed to answer generation—this prompt should flag entity coverage gaps in the output schema so reviewers can spot missing evidence. Test this prompt with a golden dataset of known entity-fact pairs to calibrate whether your retrieval pipeline is providing sufficient coverage before deduplication runs.

The entity-centric deduplication prompt operates as a pre-generation filter in your RAG pipeline. It should be invoked after retrieval but before answer synthesis, receiving the raw retrieved passages and the user query as input. The prompt's structured output—a list of retained passages with their entity-fact mappings—becomes the cleaned context window for downstream answer generation. This placement ensures that the answer model sees only one passage per unique entity-fact pair, reducing token waste and preventing the model from over-weighting repeated information while preserving entity coverage for complete answers.

To integrate this prompt into a production harness, wrap it in a validation layer that checks the output schema before passing context to the answer model. The expected output is a JSON object with a retained_passages array, where each element contains passage_id, passage_text, entity, and fact fields. Implement a schema validator that rejects responses missing required fields, containing duplicate entity-fact pairs, or exceeding the original passage count. If validation fails, retry with the same input up to two times, appending the validation error to the prompt as a correction instruction. After two failed retries, fall back to the original unfiltered context set and log the failure for investigation. Use structured logging to capture the input passage count, output passage count, deduplication ratio, validation status, and retry count for each invocation.

Model choice matters for this prompt's reliability. Use a model with strong instruction-following and structured output capabilities, such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro. Avoid smaller or older models that may collapse entity-fact distinctions or produce malformed JSON under the prompt's extraction load. For high-throughput pipelines, consider batching multiple deduplication requests or using a smaller, fine-tuned model if you have training data from production logs. Always set response_format to JSON mode when the provider supports it, and include the output schema in the prompt itself as a fallback. Monitor entity coverage loss by comparing the set of unique entities in the input versus the output—a sudden drop indicates over-aggressive deduplication that requires threshold tuning or prompt adjustment.

Observability is critical because this prompt directly shapes the evidence the answer model receives. Instrument the deduplication step with metrics: input passage count, output passage count, deduplication ratio, entity coverage preservation percentage, validation pass rate, and p95 latency. Set alerts on validation failure rates exceeding 5% and entity coverage drops below 80%. Log the full input and output for a sample of requests to enable offline debugging and prompt iteration. If your RAG pipeline serves regulated or high-stakes domains, retain these logs for audit trails that demonstrate which passages were excluded and why. The next step is to pair this deduplication prompt with a context quality gate evaluation prompt that scores the filtered context before answer generation, catching cases where deduplication removed critical evidence.