LLM Judge Deduplication Rubric Prompt

This prompt is for evaluation and quality assurance engineers who need to automate the detection of duplicate content in AI-generated outputs. Instead of relying on brittle string-matching or embedding-distance thresholds alone, this playbook provides a calibrated rubric for an LLM judge. The judge scores pairs of records, paragraphs, or claims on a defined ordinal scale, producing structured decisions that can be tracked, audited, and compared against human raters. Use this when you need a repeatable, explainable deduplication signal that fits into a regression testing suite, a data quality pipeline, or a human-in-the-loop review queue.

This is not a prompt for performing deduplication itself; it is a prompt for evaluating whether a deduplication decision is correct. If your goal is to remove duplicates from a dataset, use a sibling playbook like the Exact Duplicate Removal Prompt Template or the Near-Duplicate Paragraph Detection Prompt. This rubric is designed for the meta-evaluation layer: measuring how well your deduplication system works, calibrating thresholds, and catching edge cases where semantic similarity does not equal functional duplication. It is particularly valuable when you are tuning a fuzzy matching system, comparing the performance of multiple deduplication approaches, or building a golden test set for ongoing quality monitoring.

Do not use this prompt when you need real-time deduplication in a user-facing product flow. The pairwise comparison format introduces latency and cost that is appropriate for offline evaluation, not synchronous request paths. Also avoid this prompt when the definition of 'duplicate' is purely syntactic—simple string equality checks are faster, cheaper, and more reliable for exact matches. Reserve this rubric for cases where semantic overlap, paraphrasing, or partial redundancy create ambiguity that requires judgment. For high-stakes domains like healthcare or legal content, always include human review as a final arbiter and track inter-rater agreement between the LLM judge and domain experts before relying on automated scores.

This template provides a complete, copy-ready prompt for an LLM judge tasked with evaluating whether two records, paragraphs, or claims are duplicates. It enforces a structured JSON output, requires the judge to explain its reasoning before scoring, and uses a configurable scale with calibration examples. The prompt is designed to be pasted directly into your evaluation harness after replacing the square-bracket placeholders with your specific domain, scale, and examples.

textCopy
You are an expert data quality judge. Your task is to evaluate whether two records are duplicates according to a defined scale. You must explain your reasoning before providing a final score.

## SCALE DEFINITION
[SCALE_DEFINITION]

## CALIBRATION EXAMPLES
[EXAMPLES]

## INPUT
Record A: [RECORD_A]
Record B: [RECORD_B]

## OUTPUT SCHEMA
You must respond with a single JSON object conforming to this schema:
{
  "reasoning": "A step-by-step explanation of the evidence considered, the semantic overlap observed, and how the scale definition applies to this pair.",
  "score": [SCORE_VALUE],
  "confidence": [CONFIDENCE_VALUE]
}

## CONSTRAINTS
- Base your judgment only on the provided records and the scale definition.
- If the records are in different formats or contexts, note this in your reasoning.
- If you are uncertain, reflect that in the confidence field rather than guessing.
- Do not invent information not present in the records.

After copying the template, replace [SCALE_DEFINITION] with your duplication scale (e.g., a 1-5 Likert scale from 'Completely Distinct' to 'Identical', or a binary 'Duplicate'/'Not Duplicate' definition). Replace [EXAMPLES] with 3-5 calibrated example pairs that demonstrate boundary cases for each score level. Replace [RECORD_A] and [RECORD_B] with the actual pair to judge. [SCORE_VALUE] and [CONFIDENCE_VALUE] should be replaced with the expected types (e.g., integer or string and float between 0.0 and 1.0). For high-stakes deduplication (e.g., customer records, clinical data), always route ambiguous cases (where confidence is below your defined threshold) to a human review queue rather than auto-resolving. Validate the JSON output before accepting the score, and log the reasoning for inter-rater reliability tracking.

The deduplication rubric prompt is not a one-shot tool; it is a scoring function inside a larger evaluation harness. The harness is responsible for generating candidate pairs from your dataset, assembling the rubric prompt with the correct [RECORD_A], [RECORD_B], and [RUBRIC] placeholders, calling the judge model, and collecting structured scores. This section describes how to build that harness so you can run pairwise comparisons at scale, track judge consistency, and calibrate thresholds before promoting the rubric to production use.

Start with candidate pair generation. For a dataset of N records, avoid the O(N²) trap of comparing all pairs. Instead, use a blocking strategy: group records by a shared key such as a normalized name prefix, a locality-sensitive hash, or an embedding cosine-similarity threshold above 0.85. Within each block, generate all unique pairs. For each pair, assemble the prompt by injecting [RECORD_A] and [RECORD_B] as JSON strings with a consistent field order. The [RUBRIC] placeholder should contain the full scoring scale, definitions, and calibration examples. Call the judge model with temperature=0 and a structured output schema that enforces { "score": int, "rationale": string, "key_differences": [string] }. Validate every response: reject any output where score is outside the defined scale or where rationale is empty. On validation failure, retry once with the same prompt; if it fails again, log the pair for human review and emit a null score.

Inter-rater reliability tracking is essential because the judge model is a single rater that can drift or exhibit systematic bias. Maintain a holdout set of 50–100 human-labeled pairs with known duplicate/non-duplicate ground truth. After every scoring run, compute Cohen's kappa between the judge scores and the human labels on this holdout set. If kappa drops below 0.7, pause automated scoring and investigate: check for prompt drift, model version changes, or distribution shift in your record pairs. Log every judge decision with the pair ID, score, rationale, model version, and timestamp. This audit trail lets you trace disagreements back to specific rubric weaknesses. When you update the rubric, re-run the holdout set and compare kappa before and after the change. Do not deploy a new rubric version until holdout agreement is stable.

For production integration, wrap the harness in a queue-based worker. Enqueue candidate pairs, call the judge model with retry logic, write validated scores to a results table, and update the inter-rater dashboard. Set a maximum concurrency based on your judge model's rate limits. If you are scoring thousands of pairs, batch the holdout evaluation at the start and end of each run to detect drift without slowing throughput. The harness should also emit a summary report: total pairs scored, score distribution, validation failure rate, retry rate, and holdout kappa. This report is your release gate—if any metric crosses a predefined threshold, block the results from downstream deduplication and escalate to the evaluation owner.