Felligi-Sunter Model

The model computes a composite match weight for each record pair by summing the log-likelihood ratios of individual field comparisons. For each field, an agreement weight is assigned if values match, and a disagreement weight if they differ. The agreement weight is calculated as log2(m/u), where m is the probability that a matching pair agrees on the field, and u is the probability that a non-matching pair agrees by chance. This Bayesian foundation allows the model to handle the natural variability and errors inherent in real-world administrative data.

The model classifies record pairs into three distinct categories based on their composite weight relative to two critical thresholds:

• Upper threshold: Pairs scoring above this are classified as definitive matches.
• Lower threshold: Pairs scoring below this are classified as definitive non-matches.
• Clerical review zone: Pairs falling between the two thresholds are flagged for manual human adjudication. This triage mechanism optimizes the trade-off between automation and accuracy, ensuring high-precision linkage without losing edge cases.

A core simplifying assumption of the Felligi-Sunter model is that individual field comparisons are conditionally independent given the true match status. This means the probability of a name agreeing is assumed to be independent of whether an address agrees, for both true matches and true non-matches. While this assumption is often violated in practice—where correlated errors occur—it makes the model computationally tractable and mathematically elegant. Modern extensions often relax this assumption using log-linear models or machine learning classifiers.

The m and u probabilities are typically unknown and must be estimated from the data itself. The Expectation-Maximization (EM) algorithm is the standard unsupervised learning method for this task. It iteratively:

• E-step: Estimates the posterior probability that each pair is a match given current parameter estimates.
• M-step: Updates m and u probabilities to maximize the expected log-likelihood. This allows the model to bootstrap its own parameters without requiring a pre-labeled training set of known matches and non-matches.

The model naturally accommodates missing values by treating a blank or null field as a third comparison outcome, distinct from agreement or disagreement. A specific missing data weight is computed, reflecting the informativeness of the absence itself. For instance, a missing Social Security Number on a loan application might be a strong indicator of a non-match if the other record has one. This graceful degradation makes the model robust for real-world datasets where incomplete records are the norm, not the exception.

To improve accuracy, the model can incorporate the specific value frequency of a field. A rare name like 'Zbigniew' agreeing provides much stronger evidence of a true match than a common name like 'John' agreeing. The agreement weight is adjusted to log2(1/value_frequency), giving higher weight to rare coincidences. This refinement moves beyond simple binary agreement/disagreement to a more nuanced, information-theoretic measure of evidence, significantly reducing false positives in large-scale linkages.