Claim Deduplication and Near-Duplicate Merging Prompt

This prompt is designed for verification engineers and AI pipeline builders who need to reduce redundant claims before evidence matching to control pipeline costs and prevent duplicate work. The job-to-be-done is taking a batch of extracted claims—often from multiple documents or extraction runs—and producing a deduplicated set where semantically equivalent or near-duplicate claims are merged into a single canonical representation. The ideal user is someone operating a production verification pipeline who understands that downstream fact-checking accuracy depends on clean, non-redundant claim sets, and who needs merge provenance tracking to maintain audit trails back to original source claims.

Use this prompt when you have already extracted atomic claims from source content and need to collapse duplicates before sending them to evidence retrieval or human review. The prompt expects a structured input of claims with unique IDs, claim text, and source provenance. It outputs a deduplicated claim set with merge groups, canonical claim text, and provenance arrays showing which original claims were merged. Do not use this prompt when claims are still embedded in source documents (use an extraction prompt first), when you need to verify claims against evidence (this is a pre-verification cleanup step), or when claims contain domain-specific terminology that requires specialized equivalence rules not captured in the general deduplication criteria. The prompt is also inappropriate for merging claims that are related but semantically distinct—false merges that collapse different factual assertions will silently drop verifiable claims from your pipeline.

Before deploying this prompt, define your deduplication tolerance. Near-duplicate merging requires a judgment call about what counts as 'the same claim.' Two claims might assert the same fact with different wording, different granularity, or different numerical precision. Your prompt constraints should specify whether claims with different dates, different numerical values, or different attribution sources should be treated as duplicates or kept separate. Always implement eval checks for false merges—cases where the prompt collapses semantically distinct claims—and false duplicates—cases where it keeps separate claims that should have been merged. These evals should run against a golden dataset of claim pairs with known merge/don't-merge labels before the prompt enters production. For high-stakes verification pipelines, route merge decisions with low confidence scores to human review.

This prompt template is designed to take a list of extracted claims and identify which ones are duplicates or near-duplicates, then merge them into a canonical set. It is intended for verification pipeline engineers who need to reduce redundant claims before evidence matching to control API costs and prevent duplicate work. The template uses square-bracket placeholders that you must replace with your actual inputs, output schema, constraints, and examples before use. The prompt instructs the model to produce a deduplicated claim set where each merged claim retains a list of its source claim IDs, enabling full provenance tracking.

textCopy
You are a claim deduplication engine. Your task is to analyze a list of claims and identify duplicates and near-duplicates. Two claims are near-duplicates if they assert the same factual proposition, even if they use different wording, different granularity, or different sentence structures. Claims that are semantically distinct must NOT be merged, even if they share a topic.

[INPUT]

[EXAMPLES]

[OUTPUT_SCHEMA]

[CONSTRAINTS]

[RISK_LEVEL]

To adapt this template, replace each placeholder with concrete content. [INPUT] should contain the list of claims, each with a unique ID and the claim text. [EXAMPLES] should provide 2-4 few-shot examples showing correct deduplication decisions, including both merge and non-merge cases. [OUTPUT_SCHEMA] must specify the exact JSON structure you expect, including fields for the canonical claim text, a list of merged source claim IDs, and a confidence score for each merge decision. [CONSTRAINTS] should define rules such as 'never merge claims that differ in numerical values' or 'preserve the most specific wording when merging.' [RISK_LEVEL] should be set to high for regulated domains, which will instruct the model to flag uncertain merges for human review rather than silently merging them. After copying the template, test it against a golden dataset of known duplicates and distinct claims to calibrate merge behavior before production use.

The claim deduplication prompt is designed to sit between claim extraction and evidence matching in a verification pipeline. Its job is to reduce redundant claims before they hit expensive retrieval and LLM-based verification steps. The harness must accept a batch of extracted claims (typically as a JSON array of claim objects with IDs, text, and source provenance), pass them through the deduplication prompt, and produce a deduplicated claim set with merge provenance tracking. The prompt is not a real-time user-facing component—it runs as a batch processing step where latency tolerance is higher but correctness is critical because false merges silently drop semantically distinct claims from downstream verification.

Input assembly starts by constructing the prompt with the [INPUT_CLAIMS] placeholder populated from the upstream claim extraction output. Each claim must carry a stable claim_id, the claim_text, and any source context fields needed for deduplication reasoning. The [SIMILARITY_THRESHOLD] placeholder should be set based on your tolerance for false merges—start at 0.85 for high-recall deduplication and lower only after evaluating merge quality on a labeled dataset. The [DOMAIN_CONTEXT] placeholder should describe the subject domain (e.g., 'clinical trial results', 'financial earnings reports') to help the model distinguish domain-specific near-duplicates from genuinely distinct claims. Model choice matters: use a model with strong semantic comparison capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Smaller models often struggle with the nuanced boundary between near-duplicate and semantically distinct claims, leading to higher false-merge rates.

Output validation is the most critical harness component. The prompt outputs a JSON object with a deduplicated_claims array and a merge_log array. Before accepting the output, validate that: (1) every claim_id in the input appears exactly once across the deduplicated set and merge log combined—missing claims indicate silent drops; (2) merge log entries reference only valid input claim_id values; (3) the merged_claim_text in each deduplicated entry preserves all factual content from its source claims without introducing new assertions; (4) the merge_rationale field is non-empty and explains the semantic overlap. Implement a schema validator (Pydantic, Zod, or JSON Schema) that rejects malformed outputs and triggers a retry. For high-stakes pipelines, add a semantic drift check: embed both the original claims in a merge group and the merged output, compute cosine similarity, and flag groups below 0.9 for human review.

Retry and fallback logic should handle three failure modes. First, schema validation failures: retry up to two times with the validation error message appended to the prompt as feedback. Second, incomplete coverage: if the output doesn't account for all input claim IDs, retry with an explicit instruction listing the missing IDs. Third, low-confidence merges: if the model's self-reported confidence (if you include a [CONFIDENCE_THRESHOLD] constraint) falls below your threshold, route those specific claim groups to a human review queue rather than retrying. Human review integration is essential for any merge where the model's confidence is below 0.9 or where the semantic drift check flags a potential issue. Package the original claims, the proposed merged claim, and the merge rationale into a review task. Track reviewer decisions (accept merge, reject merge, split claims) to build an eval dataset for prompt improvement.

Logging and observability should capture: input claim count, output deduplicated count, merge group sizes, retry count, validation failures, and human review outcomes. Log the full prompt and response for any retry or human review event. This data feeds into eval runs where you compare deduplication decisions against a golden dataset of known duplicates and known distinct pairs. Key metrics: false merge rate (distinct claims incorrectly merged), false split rate (duplicates incorrectly kept separate), and merge provenance accuracy (whether the merge log correctly traces which claims were combined). Run these evals on every prompt change before deployment. Cost control is the primary motivation for this step—track the claim reduction ratio (deduplicated count / input count) and compare against the cost of running evidence matching on the full claim set to confirm the deduplication step is paying for itself.

What to avoid: Do not run deduplication on claims from different source documents unless you have a cross-document deduplication requirement—the risk of false merges increases significantly when claims lack shared context. Do not skip the merge log validation; silent claim loss is the most dangerous failure mode because it removes claims from the verification pipeline without any audit trail. Do not treat the model's merge decisions as authoritative without human spot-checking, especially in regulated domains where missed claims have compliance implications. Start with a conservative similarity threshold, build an eval dataset from production decisions, and only relax thresholds after you have measured false merge rates against ground truth.