Deduplication

A rule-based approach where two records are declared a match only if their corresponding fields are exactly equal or satisfy a predefined logical condition.

• Mechanism: Direct field-to-field comparison (e.g., SocialSecurityNumber_A == SocialSecurityNumber_B).
• Use Case: High-confidence, low-latency scenarios with a strong, immutable unique identifier.
• Limitation: Brittle against minor variations like typos, abbreviations, or whitespace differences.

Uses statistical models to calculate the likelihood that two records refer to the same entity, even when no single unique identifier exists.

• Mechanism: Weights are assigned to field agreements and disagreements (e.g., a match on 'Surname' is more significant than a match on 'City').
• Key Algorithm: Fellegi-Sunter model for record linkage.
• Output: A confidence score between 0 and 1, allowing for threshold-based decisioning.

Compares strings based on their approximate similarity rather than exact equality, tolerating typographical errors and formatting inconsistencies.

• Edit Distance: Levenshtein distance measures the number of single-character edits (insertions, deletions, substitutions) required to change one string into another.
• Phonetic Algorithms: Soundex and Metaphone index words by their pronunciation in English, catching homophones.
• Token-Based: Jaccard similarity measures overlap between sets of n-grams or words.

A performance optimization technique that avoids the O(n²) complexity of comparing every record against every other record.

• Mechanism: Partitions the dataset into mutually exclusive 'blocks' using a blocking key (e.g., first 3 letters of surname + postal code).
• Efficiency: Comparisons are only performed within each block, drastically reducing the candidate pair space.
• Trade-off: A poorly chosen blocking key can cause missed matches if the key field contains an error.

Treats deduplication as a supervised binary classification problem, where a model learns complex matching rules from labeled training data.

• Features: Similarity vectors derived from field comparisons (e.g., Jaro-Winkler score for names, numeric difference for dates).
• Algorithms: Random Forests, Gradient Boosting, or Siamese Neural Networks are commonly used.
• Advantage: Automatically discovers non-linear interactions between fields that are difficult to express with manual rules.

The final stage where matched records are consolidated into a single golden record by applying conflict resolution rules.

• Strategy: Rules define which value survives from which source (e.g., 'longest string', 'most recent timestamp', 'most trusted source').
• Lineage: Critical to preserve an audit trail mapping the golden record value back to its source record.
• Goal: Create a complete, non-redundant master profile without losing any unique information.

The process of selecting a single, authoritative identifier or record for an entity when multiple representations exist. After deduplication identifies duplicate records, canonicalization determines which version becomes the 'golden record.' This involves:

• Survivorship rules: Defining which source system's value takes precedence for each attribute (e.g., CRM overrides billing for phone numbers)
• URI selection: Choosing the canonical URL or IRI when multiple web addresses reference the same resource
• Timestamp-based resolution: Selecting the most recently updated or verified record
• Confidence-weighted merging: Combining attribute values based on source reliability scores In knowledge graphs, canonicalization consolidates authority signals—ensuring that backlinks, citations, and trust metrics aggregate on a single node rather than being diluted across duplicates.

The comprehensive framework of metrics, monitoring, and governance that ensures data remains fit for its intended use. Deduplication is one dimension within a broader quality posture that typically measures:

• Completeness: Are required fields populated?
• Consistency: Do values conform to expected formats and constraints?
• Accuracy: Do values reflect real-world truth?
• Timeliness: Is data current and available when needed?
• Uniqueness: Are duplicate records controlled? (This is where deduplication metrics live)
• Validity: Do values conform to business rules and schemas? Modern data observability platforms continuously monitor these dimensions, triggering alerts when quality thresholds are breached—preventing degraded data from poisoning downstream AI models and analytics.

The algorithmic technique for finding approximate string matches that are likely to represent the same real-world value despite typographical errors, formatting differences, or abbreviations. This is the core computational engine behind deduplication. Common algorithms include:

• Levenshtein edit distance: Minimum single-character edits (insertions, deletions, substitutions) to transform one string into another
• Jaro-Winkler similarity: Optimized for short strings like names, giving higher weight to matching prefixes
• Soundex and Metaphone: Phonetic algorithms that encode words by how they sound, catching 'Smith' vs. 'Smyth'
• n-gram similarity: Breaking strings into character subsequences and measuring overlap
• TF-IDF with cosine similarity: Treating strings as document vectors for longer text fields Advanced systems combine multiple fuzzy matchers into ensemble similarity scores and use learned distance metrics trained on domain-specific data.