Deduplication

Deduplication engines employ a spectrum of matching techniques. Exact matching identifies records where a defined key (e.g., Social Security Number) is identical. Fuzzy matching uses algorithms like Levenshtein distance or Jaro-Winkler similarity to tolerate typographical errors, transpositions, and formatting inconsistencies in fields like names and addresses.

• Deterministic rules: Simple IF-THEN logic on exact field matches
• Probabilistic scoring: Fellegi-Sunter model assigns agreement and disagreement weights
• Phonetic algorithms: Soundex or Metaphone encode names by pronunciation to catch spelling variants

Comparing every record against every other record is computationally prohibitive (O(n²) complexity). Blocking partitions the dataset into mutually exclusive buckets using a cheap heuristic called a blocking key. Only records within the same block are compared, dramatically reducing the search space.

• Sorted Neighborhood Method: Sliding a fixed-size window over sorted data
• Canopy Clustering: Using cheap distance metrics to create overlapping clusters
• TF-IDF indexing: Treating records as documents to retrieve candidate pairs via cosine similarity

Once duplicates are identified, the system must consolidate them into a single golden record. Survivorship rules define which value to retain when conflicting data exists across duplicates. Strategies range from simple source-priority ranking to complex field-level lineage tracking.

• Longest value: Retain the most complete field entry
• Most recent: Timestamp-based selection from the latest update
• Source trust: Weighted preference for authoritative systems of record
• Composite construction: Building a record from the best parts of each duplicate

A critical concept in deduplication logic: if record A matches record B, and record B matches record C, then A, B, and C must all belong to the same entity cluster. Computing this transitive closure ensures that all related duplicates are collapsed into a single group, even if A and C were never directly compared.

• Union-Find (Disjoint-Set): Efficient algorithm for dynamic connectivity
• Connected components: Graph-based clustering of the pairwise match graph
• Threshold tuning: Balancing precision and recall to avoid over-merging distinct entities

Deduplication can operate in two distinct modes. Batch deduplication runs periodically over the entire dataset to clean historical accumulation. Real-time deduplication checks incoming records against existing master data at the point of entry, preventing duplicates from being created in the first place.

• REST API integration: Synchronous match-check on form submission
• Stream processing: Kafka or Kinesis pipelines for continuous matching
• Incremental clustering: Updating existing entity groups without full recomputation

The performance of a deduplication system is measured using standard classification metrics adapted for the pairwise comparison problem. Because the number of true non-duplicate pairs vastly outweighs true duplicates, precision is often prioritized to avoid false merges that corrupt the master record.

• Precision: Fraction of predicted duplicate pairs that are true duplicates
• Recall: Fraction of all true duplicate pairs successfully identified
• F-measure: Harmonic mean balancing precision and recall
• Cluster-level metrics: B-cubed or V-measure for evaluating entity groups rather than pairs

A string comparison technique that calculates the similarity score between two text strings to identify non-exact duplicates. Tolerates typographical errors, abbreviations, and formatting inconsistencies that exact matching would miss.

• Levenshtein distance measures edit operations between strings
• Jaro-Winkler favors strings with matching prefixes
• Phonetic algorithms like Soundex match by pronunciation
• Critical for handling 'Jon Smith' vs 'John Smyth' variations

A computational efficiency technique that partitions datasets into mutually exclusive blocks using a cheap heuristic, drastically reducing the number of required pairwise comparisons from O(n²) to near-linear.

• Groups records by blocking keys like ZIP code or first letter of surname
• Only compares records within the same block
• Sorted neighborhood method slides a window over sorted data
• Without blocking, comparing 1M records requires ~500B comparisons

A statistical framework formalized by the Fellegi-Sunter model that calculates match weights for record pairs based on agreement and disagreement patterns across multiple attributes.

• Assigns log-likelihood ratios to each field comparison
• Agreement on rare values (SSN) carries more weight than common values (gender)
• Produces a composite score for classification into match, non-match, or review
• Handles the reality that no single field is perfectly reliable

The broader data management discipline of accurately matching and merging identity data across disparate systems to create a unified, 360-degree view of a customer, patient, or organization.

• Encompasses both deduplication and record linkage
• Maintains survivorship rules for conflicting attribute values
• Creates persistent global identifiers that survive system migrations
• Powers personalization engines and fraud detection systems

The process of selecting the definitive URL or entity record when multiple variants exist, consolidating authority signals into a single master reference.

• Resolves 'example.com' vs 'www.example.com' vs 'example.com/'
• Applies rel=canonical tags in web contexts
• In knowledge graphs, selects the preferred label among synonyms
• Prevents dilution of link equity and entity confidence scores