Summarization Output Deduplication Prompt

This prompt is for RAG and summarization product teams who need to clean up model-generated summaries that contain repeated claims, sentences, or facts. The core job-to-be-done is post-generation repair: you have a summary that covers the right topics but is bloated with semantic duplicates, and you need a version that is concise, non-repetitive, and preserves the original fact set. The ideal user is an engineering lead or ML engineer integrating this into a production pipeline where summaries are served to end users, stored in databases, or passed to downstream systems that penalize redundancy.

Use this prompt when a single summary response contains internal repetition—the same finding stated three ways, a key statistic repeated in multiple paragraphs, or a conclusion that echoes an earlier sentence verbatim. It is designed for intra-output deduplication, not cross-batch or cross-document deduplication. The prompt requires the full summary text and, critically, the source document or a reference fact set extracted from it. Without a ground-truth fact set, the model cannot distinguish between intentional emphasis and accidental duplication, and you risk dropping important information. Do not use this prompt for removing boilerplate or stylistic filler; that belongs in a Redundant Phrase Compression Prompt. Do not use it for deduplicating structured records or JSON arrays; use the Redundant Field Merge Prompt instead.

Before wiring this into production, define your tolerance for information loss. The prompt includes a mandatory eval step that compares the deduplicated output against the source fact set. If any fact from the source is missing in the deduplicated summary, the output should be flagged for human review or rejected. Common failure modes include over-aggressive deduplication that merges distinct but similar claims, and under-aggressive deduplication that leaves near-duplicates intact. Start with a labeled test set of 50–100 summaries with known duplicates and measure precision and recall of duplicate removal against fact retention before deploying to a live pipeline.

This prompt template is designed for RAG and summarization product teams who need to post-process model outputs that contain redundant information. It instructs the model to act as a deduplication filter, identifying and removing repeated claims, sentences, or facts from a single summary while ensuring that no unique information is lost. The template uses square-bracket placeholders that you must replace with your specific inputs, constraints, and output schema before use.

textCopy
You are an expert text deduplication engine. Your task is to analyze the provided summary and remove any repeated claims, sentences, or factual statements while preserving all unique information.

## INPUT SUMMARY
[INPUT_SUMMARY]

## SOURCE DOCUMENT FACTS (for coverage verification)
[SOURCE_FACTS]

## DEDUPLICATION RULES
- Remove verbatim repeated sentences, keeping only the first occurrence.
- Merge near-duplicate sentences that convey the same fact with different wording. Keep the most precise version.
- If two sentences partially overlap, combine them into a single sentence that captures all unique details without redundancy.
- Do not remove information that appears similar but conveys distinct facts. When in doubt, preserve the information.
- Maintain the original logical flow and paragraph structure where possible.
- Preserve all unique named entities, numbers, dates, and technical terms.

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "deduplicated_summary": "The cleaned summary text with duplicates removed.",
  "removed_items": [
    {
      "removed_text": "The text that was removed.",
      "reason": "exact_duplicate | near_duplicate | merged",
      "kept_text": "The text that was retained in its place."
    }
  ],
  "coverage_check": {
    "facts_preserved": ["List of source facts confirmed present in output"],
    "facts_missing": ["List of source facts no longer present after deduplication"],
    "coverage_score": 0.0
  },
  "compression_ratio": 0.0
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

To adapt this template, replace [INPUT_SUMMARY] with the summary text that needs deduplication. Populate [SOURCE_FACTS] with the key facts extracted from the original source document—this enables the coverage check that prevents information loss. Use [CONSTRAINTS] to add domain-specific rules, such as preserving legal terminology, maintaining citation anchors, or enforcing a maximum compression ratio. Provide [EXAMPLES] with one or two input-output pairs showing correct deduplication behavior, including edge cases where similar-but-distinct facts must be preserved. For high-stakes applications such as medical or legal summaries, add a human review step after deduplication and log all removed_items for auditability.

This prompt is designed to sit in a post-generation repair loop, not as a standalone chat interaction. After your primary summarization model produces a response, route that output through this deduplication prompt before it reaches the user, database, or downstream system. The harness must capture the original summary, the source document fact set, and any output constraints as structured inputs. Treat this as a deterministic cleanup step: the prompt expects a specific input schema and must return a specific output schema, making it suitable for programmatic validation and retry logic.

To integrate, build a thin service wrapper that accepts the raw summary and the pre-extracted source facts. The wrapper should: (1) validate that the input summary is non-empty and the fact set is a valid array of strings; (2) call the deduplication prompt with the model of your choice—GPT-4o or Claude 3.5 Sonnet work well for semantic comparison tasks; (3) parse the JSON response and validate that the deduplicated_summary field exists and that the removed_claims array contains only claims verifiably present in the original summary; (4) run an automated information-loss check by comparing the fact coverage of the deduplicated output against the original fact set using a secondary LLM judge or a simple ROUGE-L overlap metric. If coverage drops below a configurable threshold (e.g., 95%), flag for human review or fall back to the original summary. Log every deduplication decision with the original summary, deduplicated summary, removed claims, and coverage score for auditability.

Common failure modes to instrument for: the model removes a claim that appears repetitive but carries distinct information (over-aggressive dedup), the model fails to detect a true duplicate because the phrasing differs significantly (under-deduplication), or the output JSON is malformed despite schema instructions. For malformed JSON, implement a single retry with the error message fed back into the prompt. For over-aggressive dedup, tune the prompt's [CONSTRAINTS] section to emphasize preservation of distinct facts even when sentence structure is similar. For under-deduplication, lower the semantic similarity threshold described in the prompt instructions. Never send deduplicated output directly to users without the coverage check—this is the safety net that prevents information loss from reaching production.