Fellegi-Sunter Model

The foundational data structure representing the outcome of comparing two records across multiple fields. For each field i, a binary or categorical value indicates whether the values agree, disagree, or are missing.

• Binary vector: 1 for agreement, 0 for disagreement
• Example: Comparing (John, Smith, 1980) with (Jon, Smith, 1980) yields the vector [0, 1, 1]
• Missing data: A third state is often introduced to handle null values without penalizing the match score
• The vector is the sole input to the likelihood ratio calculation

The conditional probability that a field agrees given that the record pair is a true match. It quantifies the reliability of the data source.

• Formula: m = P(field agrees | true match)
• Interpretation: An m-probability of 0.95 for surname means that 95% of true matches will have the same surname
• Error source: The complement (1 - m) accounts for typographical errors, name changes, or data entry mistakes within legitimate matches
• Estimation: Typically initialized using expectation-maximization (EM) algorithms on unlabeled data

The conditional probability that a field agrees purely by random chance given that the record pair is not a true match. It represents the background noise of the dataset.

• Formula: u = P(field agrees | non-match)
• Interpretation: A u-probability of 0.001 for a rare surname means only 0.1% of non-matching pairs will coincidentally share it
• Discriminative power: Fields with low u-probabilities (like Social Security Numbers) are highly informative; fields with high u-probabilities (like gender) contribute little weight
• Estimation: Often approximated by the frequency distribution of values in the overall dataset

The ratio of the m-probability to the u-probability for a given field, expressed as a log-weight for computational stability. This is the core evidentiary unit of the model.

• Agreement weight: log₂(m/u) — positive weight supporting a match
• Disagreement weight: log₂((1-m)/(1-u)) — negative weight against a match
• Additive property: Total match weight is the sum of individual field weights, assuming conditional independence
• Example: An SSN agreement might contribute +30 to the weight, while a gender agreement contributes only +1

Two fixed cutoff values that partition record pairs into three exhaustive and mutually exclusive classes, enabling automated decision-making without manual review for the majority of pairs.

• Upper threshold (T_μ): Pairs with composite weight above this are classified as definite matches
• Lower threshold (T_λ): Pairs with composite weight below this are classified as definite non-matches
• Clerical review zone: Pairs falling between the thresholds are flagged as potential matches requiring human adjudication
• Optimal thresholds: Chosen to minimize a cost function that penalizes both false positives and false negatives

The iterative algorithm used to estimate the unknown m and u parameters from unlabeled data when no ground-truth training set of known matches exists.

• E-Step: Calculate the posterior probability that each record pair is a match, given current parameter estimates
• M-Step: Re-estimate m and u parameters by maximizing the expected log-likelihood derived from the E-step
• Convergence: The algorithm iterates until parameter changes fall below a tolerance threshold
• Latent variable: The true match status is treated as an unobserved variable, making this a classic unsupervised learning application within the model

The string comparison techniques that feed similarity scores into the Fellegi-Sunter likelihood calculations. These algorithms quantify how close two text strings are, tolerating the typographical errors and formatting inconsistencies common in real-world identity data.

• Levenshtein distance: Edit-based metric counting insertions, deletions, and substitutions
• Jaro-Winkler similarity: Optimized for short strings like names, giving higher weight to matching prefixes
• Phonetic algorithms like Soundex and Metaphone encode names by pronunciation, catching spelling variations that edit-distance metrics miss

The unsupervised machine learning step that follows pairwise matching. Once the Fellegi-Sunter model assigns match probabilities to record pairs, clustering algorithms group these pairs into disjoint identity clusters representing distinct real-world individuals.

• Transitive closure: If record A matches B, and B matches C, then A, B, and C form a single cluster
• Graph-based approaches model records as nodes and match scores as weighted edges, applying community detection algorithms
• Critical for detecting synthetic identity rings where multiple fabricated identities share overlapping attributes

Cryptographic protocols that enable the Fellegi-Sunter methodology to operate across organizational boundaries without exposing plaintext personally identifiable information. Essential for collaborative synthetic identity detection between financial institutions.

• Bloom filter encoding: Converts identity attributes into irreversible bit arrays that still support fuzzy matching via set-based similarity metrics like Dice coefficient
• Secure multi-party computation: Allows multiple parties to jointly compute match probabilities without revealing their private inputs
• Enables federated fraud detection while maintaining compliance with data privacy regulations