Canonicalization and Deduplication Combo Prompt

This playbook is for data ingestion engineers who need to clean and deduplicate model-generated records in a single repair pass. The prompt normalizes variant forms of values (dates, names, identifiers) according to configurable rules, then deduplicates records that become identical after canonicalization. Use this when a model outputs a list of entities, products, or contacts with inconsistent formatting and potential duplicates. The core job-to-be-done is taking a batch of records that are semantically equivalent but syntactically varied and collapsing them into a clean, canonical set without losing unique records.

Do not use this for semantic deduplication where records are similar but not identical after normalization. For example, if two contact records have different email addresses but likely refer to the same person, this prompt will not merge them—use the Fuzzy Deduplication with Confidence Threshold Prompt instead. This prompt assumes you have a defined set of canonicalization rules (e.g., lowercase emails, ISO 8601 dates, trimmed whitespace) and that records which normalize to the same string representation are true duplicates. It also assumes the input is a batch of records, not a streaming pipeline; for streaming, see the Duplicate Sentence Stripping Prompt Template with its streaming-compatible variant.

Before wiring this into production, ensure you have a pre/post cardinality comparison harness in place. The prompt should return both the deduplicated record set and a count of removed duplicates so you can validate that the reduction is reasonable. If the deduplication rate exceeds expected thresholds, flag the batch for human review—aggressive normalization rules can collapse records that should remain distinct. Pair this prompt with the Output Deduplication Audit Trail Prompt if you need traceable decisions for governance or compliance workflows.

This prompt performs canonicalization and deduplication as a combined repair operation. It first normalizes specified fields to a canonical form using your rules, then removes records that become identical after normalization. Use it when you need to clean ingested data where the same entity appears under different surface forms—such as inconsistent company names, address variants, or product identifiers—and you want one deduplicated, normalized result set without running two separate pipelines.

textCopy
You are a data repair system. Your task is to canonicalize and deduplicate records in a single pass.

## INPUT
[INPUT]

## CANONICALIZATION RULES
Apply these normalization rules to each record before deduplication:
[CANONICALIZATION_RULES]

## DEDUPLICATION KEY
After canonicalization, two records are duplicates if they match on these fields:
[DEDUPLICATION_KEY_FIELDS]

## CONFLICT RESOLUTION
When duplicates are found, keep the record that satisfies this rule:
[CONFLICT_RESOLUTION_RULE]

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "canonicalized_records": [
    // Array of deduplicated records after canonicalization, matching [OUTPUT_SCHEMA]
  ],
  "removed_duplicates": [
    {
      "removed_record": { /* the record that was removed */ },
      "kept_record_index": 0,
      "match_reason": "string explaining why these were duplicates"
    }
  ],
  "statistics": {
    "input_record_count": 0,
    "output_record_count": 0,
    "duplicates_removed": 0,
    "canonicalization_changes": 0
  }
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

Before returning, verify:
1. Every output record matches the canonicalization rules.
2. No two output records match on all deduplication key fields.
3. The statistics block is accurate.
4. Every removed duplicate has a clear match reason.

Adaptation guidance: Replace [INPUT] with your raw records in JSON array format. Populate [CANONICALIZATION_RULES] with field-by-field normalization instructions—for example, lowercase email addresses, strip whitespace from names, map department codes to a standard taxonomy, or normalize date formats to ISO 8601. Set [DEDUPLICATION_KEY_FIELDS] to the field combination that defines identity after normalization, such as ["normalized_email", "normalized_phone"]. Choose a [CONFLICT_RESOLUTION_RULE] like keep the record with the most recent timestamp or keep the record with the most populated fields. Add [CONSTRAINTS] for domain-specific rules such as do not merge records with different account IDs or flag but do not remove records where confidence is below 0.8. Include [EXAMPLES] showing a before/after pair with at least one normalization change and one deduplication. Set [RISK_LEVEL] to low, medium, or high to control downstream behavior—high-risk workflows should route removed duplicates to a human review queue rather than discarding them silently.

Wire this prompt into a repair pipeline that runs after initial extraction or generation. Validate the output JSON against the schema before accepting it. Compare statistics.input_record_count against your actual input count to detect truncation. For high-risk data, log every removed_duplicates entry to an audit table and require human approval before the deduplicated set replaces source-of-truth records. Avoid using this prompt when records have semantic identity that cannot be captured by field-level canonicalization rules alone—those cases need an LLM judge deduplication rubric instead.

This prompt is designed to operate as a single-pass repair step inside a data ingestion or batch processing pipeline. It expects a list of input records and a set of canonicalization rules, and it returns a deduplicated list where variant forms have been normalized before comparison. The harness must enforce that the output cardinality is less than or equal to the input cardinality, and that no record was dropped unless it became identical to another record after canonicalization.

Wire the prompt into your pipeline as a post-extraction, pre-load transform. The typical flow is: raw source → extraction prompt → schema validation → canonicalization + dedup prompt → final schema validation → load. Pass [INPUT_RECORDS] as a JSON array of objects, [CANONICALIZATION_RULES] as a map of field-to-rule entries (e.g., {"email": "lowercase_trim", "phone": "e164_format"}), and [OUTPUT_SCHEMA] as the expected record shape. The model should return a JSON object with canonicalized_records, removed_duplicates, and cardinality_report keys. Validate the response immediately: confirm len(canonicalized_records) + len(removed_duplicates) == len(input_records), and that every removed duplicate has a canonical_match_id pointing to a surviving record.

For production resilience, implement a retry wrapper around the prompt call. If the output fails structural validation (missing keys, wrong types, cardinality mismatch), retry up to two times with the validation error message appended to the prompt as [PREVIOUS_ERROR]. If retries are exhausted, route the batch to a human review queue with the input, the failed outputs, and the validation errors logged. Use a deterministic canonicalization library (e.g., a phone number parser or email normalizer) to pre-process fields where possible before the prompt runs—this reduces model variance and token cost. Log the cardinality_report for every batch to track deduplication rates over time and detect drift in input data quality. Avoid using this prompt for real-time, single-record flows; it is optimized for batch sizes between 10 and 200 records where cross-record comparison is valuable.