Fellegi-Sunter Model

The model assigns a log-likelihood ratio to each field comparison. Agreement on a field contributes a positive weight proportional to the log of m/u, where m-probability is the chance that matching records agree on the field, and u-probability is the chance that random records agree. Disagreement contributes a negative weight based on (1-m)/(1-u). The total composite weight is the sum of individual field weights, providing a single scalar score for classification.

Rather than forcing a binary match/non-match decision, the model partitions record pairs into three regions using two thresholds:

• Upper threshold: Pairs scoring above this are classified as definitive matches
• Lower threshold: Pairs scoring below this are classified as definitive non-matches
• Clerical review region: Pairs between the thresholds are flagged as potential matches requiring human adjudication This triage mechanism minimizes both false positives and false negatives in high-stakes identity resolution.

The classic Fellegi-Sunter formulation assumes that identity fields are conditionally independent given the true match status. This means that knowing whether two records agree on surname does not influence the probability of agreement on address, once you know if they represent the same person. While often violated in practice—correlated fields like city and ZIP code—this simplification makes the model computationally tractable and remains surprisingly robust for most linkage tasks.

When ground-truth training data is unavailable, the m and u probabilities are estimated directly from the data using the Expectation-Maximization (EM) algorithm:

• E-step: Estimate the probability that each pair is a match given current parameter estimates
• M-step: Update m and u probabilities to maximize the expected log-likelihood This unsupervised learning approach allows the model to bootstrap itself from raw, unlabeled record pairs, making it ideal for cold-start identity resolution scenarios.

Applying the Fellegi-Sunter model to all possible record pairs is computationally prohibitive—a Cartesian product of two datasets with 1 million records each yields 1 trillion comparisons. Blocking reduces this by partitioning records into mutually exclusive blocks using high-confidence fields like ZIP code or Soundex-encoded surname. Only pairs within the same block are scored, reducing complexity from O(n²) to near-linear while preserving recall.

Not all identity fields are equally reliable. The model naturally handles this by assigning higher agreement weights to discriminating fields with low u-probabilities—like a Social Security Number where random agreement is 10⁻⁹—and lower weights to common values like birth year. This intrinsic reliability weighting means the model automatically learns which fields are most diagnostic for identity resolution without requiring manual feature engineering or domain expertise.

The counterpart to probabilistic linkage that relies on exact, verified matches of personally identifiable information (PII). Where Fellegi-Sunter calculates match weights from multiple partial agreements, deterministic matching requires absolute field equality—such as a hashed email or login credential—to link records with 100% confidence. The two approaches are complementary: deterministic rules provide high-precision anchors, while Fellegi-Sunter resolves the ambiguous long tail of non-exact matches.

The broader category of statistical identity resolution that the Fellegi-Sunter model formalized. Probabilistic matching uses non-personal signals—IP addresses, browser fingerprints, device types, and behavioral patterns—to infer device ownership. Key distinctions from Fellegi-Sunter's classical formulation:

• Modern implementations incorporate temporal decay on linkage weights
• Supervised ML variants learn agreement/disagreement weights from labeled training data
• Real-time systems apply streaming EM algorithms to update m and u probabilities continuously

The centralized data structure that stores the output of Fellegi-Sunter linkage decisions. An identity graph links all known identifiers—email addresses, device IDs, usernames, and household keys—to a single canonical customer profile. The graph materializes the match/non-match/potential-match classifications:

• Match edges connect identifiers with high composite weights above the upper threshold
• Potential-match edges remain pending manual review or additional signal collection
• Non-match edges are explicitly severed to prevent false merges

The critical KPI measuring the percentage of user records successfully linked between two disparate datasets. In Fellegi-Sunter terms, match rate reflects the proportion of record pairs whose composite weight exceeds the upper threshold. Factors degrading match rate:

• Sparse overlap in available blocking keys between datasets
• Data quality issues like missing fields or inconsistent formatting
• Overly conservative thresholds that minimize false positives at the expense of recall
• Identity decay where stale identifiers no longer produce agreement on current attributes

An algorithmic technique that identifies non-identical but similar text strings using edit distance metrics, directly feeding into Fellegi-Sunter's agreement probability calculations. Rather than binary agree/disagree, fuzzy matching produces a continuous similarity score (0-1) that replaces the simple agreement indicator. Common metrics:

• Levenshtein distance: minimum single-character edits between strings
• Jaro-Winkler: optimized for name matching with prefix weighting
• Jaccard similarity: token-based overlap for multi-word fields like addresses
• Phonetic encoding (Soundex, Metaphone): captures pronunciation similarity for names

A secure, neutral environment where multiple parties combine first-party datasets for identity resolution without exposing raw user-level data. Fellegi-Sunter provides the mathematical foundation for privacy-preserving record linkage within clean rooms:

• Parties compute agreement patterns on hashed or encrypted identifiers
• Match weights are calculated without revealing which specific fields agreed
• Only the final match/non-match classification crosses organizational boundaries
• Differential privacy noise can be injected into the composite weight distribution to prevent inference attacks on individual records