Fellegi-Sunter Model

The core input to the model is a comparison vector (γ) for each record pair. This vector encodes the agreement or disagreement across a set of comparison fields (e.g., name, date of birth, address).

• For each field, a comparison is made (e.g., exact match, partial match using Jaccard similarity, or phonetic match).
• The resulting vector γ is a binary or multi-level representation of the agreement pattern. For example, γ = (1, 0, 1) could indicate agreement on name and address, but disagreement on date of birth.
• The model's entire probabilistic calculation is conditioned on this observed comparison vector.

The model is built on two fundamental conditional probability distributions:

• m-probabilities: The probability of observing a specific agreement pattern γ given that the two records are a true match (belong to the same entity). Formally: m(γ) = P(γ | M).
• u-probabilities: The probability of observing γ given that the two records are a true non-match (belong to different entities). Formally: u(γ) = P(γ | U).

These probabilities are estimated from training data or using the Expectation-Maximization (EM) algorithm. High m-probabilities indicate an agreement is highly indicative of a match, while high u-probabilities indicate an agreement is common by chance.

For each observed agreement pattern γ, the model computes a composite weight or log-likelihood ratio. This is the core scoring mechanism.

Weight W_γ = log2( m(γ) / u(γ) )

• A positive weight adds evidence for a match (m-probability > u-probability).
• A negative weight adds evidence for a non-match (u-probability > m-probability).
• A weight near zero is neutral.
• The total weight for a record pair is the sum of the weights for each individual field's agreement/disagreement, assuming conditional independence. This total score quantifies the combined evidence.

The final linkage decision is made by comparing the total composite weight (W) to two pre-defined thresholds.

• Upper Threshold (T_μ): If W ≥ T_μ, declare the pair a match.
• Lower Threshold (T_λ): If W ≤ T_λ, declare the pair a non-match.
• If T_λ < W < T_μ, the pair is placed in a clerical review pool for human adjudication.

These thresholds are set based on desired error tolerances, balancing false matches (Type I error) and false non-matches (Type II error). The region between them explicitly acknowledges uncertainty.

A key innovation was using the Expectation-Maximization algorithm to estimate m- and u-probabilities from unlabeled data.

1. Initialization: Start with plausible guesses for m- and u-probabilities.
2. Expectation Step (E-Step): Use current probabilities to calculate the likelihood that each record pair is a match, given its comparison vector γ.
3. Maximization Step (M-Step): Update the m- and u-probabilities using the match likelihoods from the E-step as weights.
4. Iteration: Repeat E and M steps until the probability estimates converge.

This allows the model to be trained without a fully labeled set of matches and non-matches, which is often impractical to obtain.

A foundational simplification in the classic Fellegi-Sunter model is the assumption that field comparisons are conditionally independent, given the match status (M or U).

• This means: P(γ | M) = Π P(γ_i | M) and P(γ | U) = Π P(γ_i | U).
• It allows the composite weight to be a simple sum of per-field weights.
• In practice, this assumption is often violated (e.g., city and zip code are correlated). Extensions to the model, such as using Bayesian networks or dependency graphs, have been developed to account for these correlations, making the probability estimation more accurate.

The model is used to create a Single Customer View by linking customer records from CRM, e-commerce, support tickets, and marketing platforms. It probabilistically determines if 'John Smith' in Salesforce and 'J. Smith' in the billing system are the same person.

• Key Challenge: Handles variations in name formatting, address typos, and missing fields.
• Business Impact: Enables unified analytics, personalized marketing, and accurate lifetime value calculation.

Critical for patient safety and operational efficiency, the Fellegi-Sunter model links patient records across hospitals, clinics, and labs to create a definitive Master Patient Index.

• Key Challenge: Resolves matches despite inconsistent identifiers (e.g., SSN not always collected), name changes, and data entry errors.
• Probabilistic Weights: Attributes like Date of Birth, Medical Record Number, and Address are assigned match/non-match probabilities to avoid dangerous false merges.

Used in Know Your Customer (KYC) and anti-money laundering (AML) processes to identify if a new account applicant matches any entity on sanctions lists or is a known risky actor across global databases.

• Application: Calculates the likelihood that 'Mohammed Al-Sayed' from one watchlist is the same as 'M. Al Sayed' in a transaction log.
• Regulatory Need: Provides an auditable, statistical basis for match decisions, which is essential for regulatory compliance.

The original and canonical application. Used by statistical agencies to deduplicate census responses, link historical records for longitudinal studies, and integrate survey data with administrative records.

• Historical Context: Ivan Fellegi and Alan Sunter developed the model at Statistics Canada to improve population count accuracy.
• Scale: Efficiently processes hundreds of millions of record pairs using blocking strategies to make comparisons computationally feasible.

Creates a clean Vendor Master by resolving duplicate and conflicting supplier entries from ERP, procurement, and invoicing systems. Determines if 'Intel Corp.', 'Intel Corporation', and 'Intel Inc.' refer to the same legal entity.

• Business Driver: Prevents duplicate payments, ensures contract compliance, and enables accurate spend analysis.
• Data Sources: Matches on company name, DUNS number, tax ID, and address with appropriate probabilities for each field.

Applied in digital libraries and research analytics to link author names across millions of publications, solving the author name disambiguation problem (e.g., is 'J. Smith' the biologist or the computer scientist?).

• Methodology: Uses co-author networks, institutional affiliations, research topics, and publication venues as matching attributes within the probabilistic framework.
• Output: Enables accurate citation analysis, h-index calculation, and research network mapping.

Record linkage is the foundational task of identifying records across one or more datasets that correspond to the same real-world entity. It is the primary problem the Fellegi-Sunter model was designed to solve.

• Purpose: Enables data integration, deduplication, and the creation of unified views from disparate sources.
• Scope: Can be performed within a single dataset (deduplication) or across multiple datasets (entity resolution).
• Methods: Encompasses both deterministic (rule-based) and probabilistic (model-based, like Fellegi-Sunter) approaches.

Probabilistic matching is a statistical approach to entity resolution that calculates the likelihood that two records refer to the same entity, rather than relying on exact, deterministic rules.

• Core Mechanism: Uses models to weigh the evidence provided by agreements and disagreements on various record attributes (e.g., name, address, date of birth).
• Advantage over Deterministic: More robust to data errors, typos, and missing values. It can handle partial matches and assign confidence scores.
• Fellegi-Sunter's Role: Provides the canonical mathematical framework for calculating match (m-probability) and non-match (u-probability) probabilities for attribute comparisons.

The Expectation-Maximization algorithm is an iterative optimization method crucial for training the Fellegi-Sunter model when true match status is unknown.

• Function: Used to estimate the m-probabilities and u-probabilities for each attribute from unlabeled record pair data.
• Process:
  • E-step: Calculates the expected probability that each record pair is a match, given current parameter estimates.
  • M-step: Updates the m and u probabilities to maximize the likelihood of the observed agreement patterns.
• Outcome: Converges on the optimal statistical parameters for the linkage model without requiring a fully labeled training set.

Blocking is a pre-processing technique used to make large-scale entity resolution computationally feasible by reducing the number of candidate record pairs for comparison.

• The Problem: Comparing every record against every other record (an O(n²) operation) is intractable for large datasets.
• The Solution: Records are partitioned into blocks based on a common, relatively error-free key (e.g., postal code, phonetic name code). Only records within the same block are compared.
• Trade-off: Introduces a risk of missed matches if the blocking key is incorrect or missing (blocking error). Advanced methods like multi-pass blocking or locality-sensitive hashing (LSH) mitigate this.

Precision and recall are the core evaluation metrics for any entity resolution system, including those built on the Fellegi-Sunter model. They quantify the trade-off between accuracy and completeness.

• Precision: The fraction of declared matches that are correct. High precision means few false positives (incorrectly linked records).
• Recall: The fraction of all true matches in the data that are successfully found. High recall means few false negatives (missed matches).
• Fellegi-Sunter Application: The model's decision rule involves setting threshold weights. A high threshold favors precision; a low threshold favors recall. The ROC curve visualizes this trade-off for different thresholds.

A golden record is the single, authoritative, and consolidated representation of an entity created after the entity resolution process is complete.

• Purpose: Serves as the "source of truth" for an entity (e.g., a customer, product, or supplier) across the enterprise.
• Creation: Produced by merging and surviving data from all source records that were linked as referring to the same entity. This involves resolving conflicts (e.g., two different addresses) through rules or survivorship functions.
• Relationship to Fellegi-Sunter: The probabilistic linkage model identifies which records should be merged. The golden record defines what the final, clean output of that merge looks like.