Record Linkage

The Fellegi-Sunter model provides the mathematical foundation for probabilistic record linkage. It calculates match weights for record pairs based on agreement and disagreement patterns across multiple attributes.

• M-Probability: The likelihood that a matching field agrees, given the pair is a true match
• U-Probability: The likelihood that a matching field agrees, given the pair is a false match
• Composite Weight: Sum of log-likelihood ratios across all fields, used to classify pairs as matches, non-matches, or requiring clerical review

Pairwise comparison of all records across datasets is computationally infeasible at scale. Blocking partitions datasets into mutually exclusive blocks using cheap heuristics, drastically reducing comparisons.

• Phonetic Blocking: Groups records by Soundex or Metaphone encodings of names
• Sorted Neighborhood: Sorts records by a key and slides a fixed-size window to compare only nearby records
• Canopy Clustering: Uses inexpensive distance metrics to create overlapping clusters for candidate generation

Exact matching fails against typographical errors, abbreviations, and formatting inconsistencies. Fuzzy matching algorithms quantify string similarity to tolerate real-world data quality issues.

• Levenshtein Distance: Minimum number of single-character edits required to transform one string into another
• Jaro-Winkler: Optimized for short strings like names, giving higher scores to strings that match from the beginning
• TF-IDF Cosine Similarity: Treats strings as token vectors, effective for longer fields like addresses

Record linkage strategies fall on a spectrum from rigid rule-based systems to flexible statistical models.

• Deterministic Linkage: Requires exact or partial agreement on a predefined set of identifiers. Simple and auditable but brittle against data quality issues
• Probabilistic Linkage: Uses statistical models to estimate match probability, tolerating disagreement on some fields while still identifying true matches
• Hybrid Models: Apply deterministic rules for high-confidence matches and probabilistic scoring for ambiguous cases, balancing precision and recall

When linking records containing personally identifiable information across organizations, PPRL techniques enable matching without revealing plaintext data to any party.

• Bloom Filter Encoding: Converts identifiers into bit arrays using multiple hash functions, enabling approximate matching on encrypted representations
• Phonetic Encoding: Transforms names into privacy-preserving phonetic codes before comparison
• Secure Multiparty Computation: Distributes the linkage computation across parties so no single entity sees the complete data

Rigorous evaluation is essential because linkage errors propagate into downstream analyses. Key metrics include:

• Precision: Proportion of declared matches that are true matches
• Recall: Proportion of true matches successfully identified by the algorithm
• F-Measure: Harmonic mean of precision and recall, providing a single quality score
• False Match Rate: Critical in healthcare and finance, where a single false link can have severe consequences

Hospitals use probabilistic record linkage to merge patient records from disparate electronic health record (EHR) systems, laboratories, and pharmacies. By matching on attributes like name, date of birth, and ZIP code without a universal patient ID, an Enterprise Master Patient Index (EMPI) prevents duplicate medical records, reduces the risk of dangerous medication errors, and ensures a longitudinal view of a patient's history. Fuzzy matching is critical here to account for typos, hyphenated names, and transposed digits in manual data entry.

Intelligence agencies link watchlists, travel manifests, and financial transactions to identify bad actors operating under multiple aliases. Identity resolution systems perform cross-document coreference across classified and open-source intelligence (OSINT) to connect a single individual to multiple passport numbers, phone numbers, and addresses. This application demands high-precision deterministic matching on biometrics combined with probabilistic scoring on transliterated names to avoid false positives that restrict innocent travelers.

Marketplaces aggregate product feeds from thousands of sellers, resulting in millions of SKUs where the same product is listed with slightly different titles and descriptions. Deduplication pipelines use TF-IDF vectorization and entity embeddings to cluster identical products into a canonical listing. This improves search relevance, prevents price fragmentation, and enables accurate competitive pricing analysis. A 'Nikon D850 DSLR Camera' and 'Nikon D850 Digital SLR' must resolve to the same product entity despite differing surface forms.

Financial institutions are mandated to screen customer databases against sanctions lists from OFAC, the UN, and the EU. This requires high-speed candidate generation to match customer names against politically exposed persons (PEPs) and sanctioned entities. Transliteration models handle Cyrillic or Arabic script names, while acronym expansion resolves corporate entity variations. The linkage must be auditable to satisfy regulatory scrutiny, balancing the risk of missing a true match against the operational cost of investigating false positives.

A string comparison technique that calculates the similarity between two text strings to identify non-exact duplicates, tolerating typographical errors, abbreviations, and formatting inconsistencies. Core algorithms include:

• Levenshtein distance: Minimum number of single-character edits required to transform one string into another.
• Jaro-Winkler distance: Optimized for short strings like names, giving higher scores to strings that match from the beginning.
• Jaccard similarity: Measures overlap between sets of n-grams, effective for comparing multi-word addresses.

The specific application of record linkage techniques to find and merge duplicate records representing the same entity within a single dataset to create a clean master record. Unlike general record linkage, which often joins two separate files, deduplication is an internal housekeeping operation. Common scenarios include:

• Cleaning a CRM with multiple entries for the same contact.
• Consolidating patient records in an EHR system.
• Removing redundant product SKUs from an inventory database. The output is a golden record that synthesizes the most complete and accurate attributes from all duplicate instances.