Batch Processing Claim Deduplication Prompt Template

This prompt is for pipeline engineers who need to prevent redundant verification work by identifying and merging near-duplicate claims before they hit expensive evidence retrieval and matching stages. The job-to-be-done is cost control and processing efficiency: when a batch of extracted claims contains semantically equivalent statements with minor wording differences, running each through the full verification pipeline wastes compute, API budget, and human review time. The ideal user is a platform engineer or verification pipeline operator who already has a claim extraction step producing structured claim objects and needs a deduplication stage that produces a canonical claim set plus a mapping of duplicates to their canonical representatives.

Use this prompt when your claim volume exceeds a few hundred items per batch and you observe that 15-40% of claims are near-duplicates of each other—common in multi-document extraction runs where the same fact appears in slightly different phrasings across sources. The prompt expects structured input: each claim must have a unique ID, the claim text, and optionally its source document reference. Do not use this prompt for exact string matching (a simple hash or text comparison is cheaper and more reliable) or for claims that differ in factual content rather than phrasing—semantic deduplication should merge 'The CEO resigned in March 2024' and 'The chief executive stepped down last March' but must not merge 'Revenue grew 12%' and 'Revenue grew 20%' even if they share structure.

Before deploying this prompt into production, you must implement a pre-filter for exact duplicates using normalized text hashing, and a post-validation step that checks the deduplication output for false merges—cases where claims with different factual content were incorrectly grouped. The prompt is designed for a single model call per batch, but for batches exceeding 50 claims, you should split into chunks of 30-50 claims with overlap windows to avoid missing cross-chunk duplicates. Always log the duplicate mappings for auditability, and route any claims where the model's confidence in a merge decision is below your threshold to a human review queue before the canonical claim proceeds to evidence retrieval.

This prompt template is designed for pipeline engineers who need to prevent redundant verification of near-duplicate claims. It takes a batch of extracted claims and identifies semantic duplicates, producing a canonical claim set with explicit duplicate-to-canonical mappings. The template uses square-bracket placeholders so you can wire it into your verification pipeline with your own claim schemas, deduplication thresholds, and output contracts.

textCopy
You are a claim deduplication engine in a fact-checking pipeline. Your task is to process a batch of extracted claims, identify semantic near-duplicates, and produce a canonical claim set with explicit duplicate mappings.

## INPUT
[BATCH_INPUT]

## DEDUPLICATION RULES
1. Two claims are duplicates if they assert the same factual proposition, even if wording, granularity, or framing differs.
2. Claims that are more specific versions of a general claim should be treated as duplicates of the more specific claim, not the general one.
3. Claims about different entities, time periods, locations, or numerical values are NOT duplicates, even if structurally similar.
4. Claims that differ in modality (e.g., "X happened" vs. "X may have happened") are NOT duplicates.
5. Claims that differ in attribution (e.g., "Source A says X" vs. "Source B says X") are NOT duplicates if the attribution is part of the claim.

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "canonical_claims": [
    {
      "canonical_id": "string",
      "canonical_text": "string",
      "original_claim_ids": ["string"],
      "claim_count": integer,
      "representative_claim_id": "string"
    }
  ],
  "duplicate_mappings": [
    {
      "original_claim_id": "string",
      "canonical_id": "string",
      "similarity_notes": "string"
    }
  ],
  "singleton_claims": ["string"],
  "merge_statistics": {
    "input_claim_count": integer,
    "canonical_claim_count": integer,
    "duplicate_claim_count": integer,
    "singleton_claim_count": integer,
    "largest_merge_group_size": integer
  },
  "uncertain_merges": [
    {
      "claim_id_a": "string",
      "claim_id_b": "string",
      "uncertainty_reason": "string",
      "recommendation": "manual_review | auto_merge | keep_separate"
    }
  ]
}

## CONSTRAINTS
- Every input claim must appear in exactly one canonical group or as a singleton.
- Canonical text should be the clearest, most complete version among the duplicates.
- If unsure whether two claims are duplicates, flag them in `uncertain_merges` rather than forcing a decision.
- Do not invent claims. Every canonical claim must be traceable to at least one input claim.
- Preserve all original claim IDs for audit trail purposes.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Process the batch and return only the JSON output.

To adapt this template, replace [BATCH_INPUT] with your serialized claim batch—typically a JSON array of claim objects with at minimum claim_id and claim_text fields. Provide [FEW_SHOT_EXAMPLES] with 2-4 annotated examples showing clear duplicates, near-duplicates, and non-duplicates that are structurally similar. Set [RISK_LEVEL] to high if downstream verification actions are irreversible, which should trigger stricter merge thresholds and mandatory human review for uncertain_merges. For production use, add a [CONFIDENCE_THRESHOLD] parameter to control the boundary between auto-merge and flag-for-review, and consider adding a [DOMAIN_CONTEXT] field if your claims use domain-specific terminology that affects semantic equivalence judgments.

Before deploying this prompt into a batch pipeline, validate that your input schema matches the expected claim structure, test with known duplicate sets to measure merge precision and recall, and implement a post-processing validator that checks the output JSON for completeness—every input claim ID must appear in either canonical_claims, duplicate_mappings, or singleton_claims. For high-stakes verification pipelines, route all uncertain_merges to a human review queue and log merge decisions for audit trail assembly.

The deduplication prompt is designed to operate as a stateless batch processor within a larger verification pipeline. It receives a batch of claims, produces a canonical claim set with duplicate mappings, and returns structured output that downstream stages consume. The implementation harness must handle input batching, output validation, idempotency, and failure recovery. Because deduplication errors can cascade—either by merging distinct claims (false positives) or by treating near-duplicates as separate (false negatives)—the harness must include strict validation gates before the deduplicated set proceeds to evidence retrieval or scoring stages.

Input assembly requires grouping claims into batches sized for the model's context window and latency budget. For a typical 8K–32K context model, batch sizes of 50–200 claims work well, depending on average claim length. Each batch should include a unique batch_id, a run_id for the pipeline execution, and the full claim objects with their source provenance. The harness should deduplicate claims within a batch first, then optionally run a second pass across batch boundaries for high-recall use cases. Output validation must check: (1) every input claim appears in exactly one canonical group or the duplicate mapping, (2) canonical claim IDs are stable and unique, (3) no claim is mapped to itself as a duplicate, and (4) the output schema matches the expected contract. A validation failure should trigger a retry with the same batch and an error context appended to the prompt, not silently pass malformed output downstream.

Retry and recovery logic should distinguish between transient failures (malformed JSON, partial output) and persistent failures (consistent semantic errors, repeated false merges). For transient failures, implement up to three retries with exponential backoff, appending the validation error message to the prompt on each retry. For persistent failures, route the batch to a dead-letter queue with the full input, output, and validation trace for human review. Model selection matters: smaller models may struggle with subtle semantic distinctions between claims that differ only in quantification, temporal scope, or attribution. Use a capable model (GPT-4-level or equivalent) for the deduplication step, and consider running a cheaper model as a consistency check on a sample of outputs. Observability requires logging: batch size, input claim count, output canonical count, duplicate count, merge ratio, validation pass/fail, retry count, and latency. Set an alert if the merge ratio exceeds 0.9 (suggesting over-aggressive deduplication) or falls below 0.05 on known-duplicate-heavy batches (suggesting under-deduplication).

Integration points with the broader pipeline: the deduplication stage should sit immediately after claim extraction and before evidence retrieval. This prevents redundant retrieval calls for near-duplicate claims, directly reducing cost and latency. The canonical claim set should preserve a source_claim_ids array linking back to original claims for audit trail purposes. When a downstream verification stage produces a verdict for a canonical claim, the harness must propagate that verdict to all duplicate claims in the mapping. Human review gates are essential when the model flags claims as 'uncertain' or when the merge confidence falls below a configurable threshold (default: 0.7). Package uncertain cases with both claims, the model's reasoning, and a structured review prompt for a human operator. Do not auto-merge uncertain pairs without review, as false merges are harder to detect and correct downstream than false duplicates.