Entity Resolution

The core logic engine distinguishing between rigid rules and statistical likelihood.

• Deterministic Linkage: Requires exact agreement on a predefined set of identifiers (e.g., SSN + DOB). It offers high precision but fails catastrophically with typos or missing data.
• Probabilistic Linkage: Uses the Fellegi-Sunter model to calculate match weights based on field agreement/disagreement patterns. It tolerates noise by estimating the probability that two records belong to the same entity, optimizing for recall in dirty datasets.

Avoiding the quadratic trap of comparing every record to every other record.

Comparing 100 million records naively requires 5 × 10¹⁵ comparisons. Blocking partitions the dataset into mutually exclusive clusters using a Blocking Key (e.g., Zip Code + First Letter of Last Name).

• Sorted Neighborhood Method: Slides a fixed-size window over sorted data to compare only nearby records.
• TF-IDF Blocking: Uses term frequency to select the most discriminative tokens for indexing, preventing overly large blocks from common values.

Quantifying the distance between imperfect textual representations of the same name or address.

• Edit Distance (Levenshtein): Counts the minimum insertions, deletions, or substitutions to transform one string into another.
• Jaro-Winkler Distance: Optimized for short strings like personal names; gives higher scores to strings with matching prefixes to account for common typographical errors.
• Phonetic Encoding (Soundex/Double Metaphone): Indexes words by pronunciation to match homophones (e.g., 'John' vs. 'Jon'), resolving spelling variations in spoken names.

Merging a cluster of matched records into a single source of truth.

Once a Transitive Closure identifies all records belonging to the same entity, survivorship rules resolve conflicting attribute values. A Golden Record is created by applying rules such as:

• Recency: Selecting the most recently updated value.
• Completeness: Preferring the longest or most filled-in attribute.
• Source Trust: Prioritizing data from the most authoritative system. This process is the cornerstone of Master Data Management (MDM).

Measuring the accuracy of the resolution pipeline against ground truth.

Without quality metrics, resolution is blind. Evaluation relies on:

• Precision: The fraction of declared matches that are true matches.
• Recall: The fraction of true matches successfully found by the system.
• F-Measure: The harmonic mean balancing precision and recall.
• False Match Rate: The proportion of non-matching pairs incorrectly linked, a critical error metric in high-stakes domains like healthcare.

The critical preprocessing phase that normalizes heterogeneous inputs.

Before matching, data must be transformed into a canonical format. Schema Alignment maps semantically equivalent attributes (e.g., 'CustName' to 'FullName'). Data Standardization parses and cleans values:

• Splitting '123 Main St, NY 10001' into atomic street, city, and zip fields.
• Normalizing date formats (MM/DD/YYYY vs. DD-MM-YY).
• Removing stop words and punctuation to reduce noise in comparison vectors.

The two fundamental paradigms for matching records. Deterministic linkage requires exact agreement on a predefined set of identifiers, offering high precision but failing on typographical errors. Probabilistic linkage, grounded in the Fellegi-Sunter model, uses statistical likelihood ratios to compute match weights, accounting for data quality issues and partial agreement. Key distinctions:

• Deterministic: Rule-based, rigid, high precision, low recall
• Probabilistic: Data-driven, flexible, balances precision and recall
• Probabilistic models naturally handle missing values and frequency-adjusted weights

Techniques to reduce the quadratic complexity of pairwise comparison from O(n²) to near-linear time. Blocking key selection partitions datasets into mutually exclusive blocks using attributes like zip code or birth year. The Sorted Neighborhood Method slides a fixed-size window over sorted records. Advanced approaches include TF-IDF blocking, which weights tokens by discriminative power, and Locality-Sensitive Hashing (LSH), which hashes similar items into the same bucket with high probability for privacy-preserving applications.

Algorithms that quantify the similarity between strings to overcome typographical errors and formatting inconsistencies. Edit distance thresholds (Levenshtein, Damerau-Levenshtein) measure the minimum character operations required to transform one string into another. Jaro-Winkler distance is optimized for short strings like personal names, giving higher scores to matching prefixes. Phonetic encoding (Soundex, Double Metaphone) indexes words by pronunciation to match homophones, critical for linking names with spelling variations across systems.

The governance layer that consumes entity resolution output. A golden record is the single best-curated version of a master data entity, created by resolving and merging conflicting attributes through survivorship rules. Master Data Management (MDM) provides the comprehensive framework ensuring uniformity, accuracy, and semantic consistency of shared critical data assets. MDM defines:

• Survivorship strategies for conflicting field values
• Data stewardship workflows for ongoing curation
• Semantic consistency rules across source systems

Rigorous evaluation of entity resolution output using precision, recall, and F-measure against ground-truth data. False match rate measures the proportion of non-matching pairs incorrectly linked, while false non-match rate captures missed true matches. Clerical review handles record pairs in the uncertainty region where automated scoring cannot confidently classify. Transitive closure applies graph-based resolution to identify all connected components in a pairwise comparison graph, ensuring consistent entity clustering across the entire dataset.

A cryptographic framework enabling entity resolution across disparate databases without revealing plaintext identifiers of non-matching records. Bloom filter encoding hashes sensitive identifiers into bit-arrays for approximate matching. Cryptographic Longterm Keys (CLK) generate irreversible, salted hash-based tokens. Private blocking uses techniques like LSH to reduce comparisons without leaking similarity. Secure multi-party computation protocols allow two or more parties to jointly compute matches without exposing individual-level data to any participant.