Probabilistic Linkage

The core of the Felligi-Sunter model, this mechanism assigns a numerical weight to each identifier field based on its agreement or disagreement. A match weight is calculated as the log-likelihood ratio, comparing the probability of observing the data pattern if the records are a true match (m-probability) versus if they are a random coincidence (u-probability).

• Agreement Weight: Positive contribution when fields match (e.g., same rare last name).
• Disagreement Weight: Negative contribution when fields conflict (e.g., different years of birth).
• Composite Score: The sum of all field weights determines the final match probability.

Unlike deterministic rules, probabilistic linkage accounts for real-world data quality issues by comparing fields using approximate string metrics rather than strict equality. This allows the system to recognize that 'Catherine' and 'Katherine' likely refer to the same individual.

• Edit Distance: Quantifies the number of single-character edits (insertions, deletions, substitutions) required to transform one string into another.
• Phonetic Encoding: Algorithms like Soundex and Double Metaphone index names by their pronunciation to match homophones.
• Jaro-Winkler Distance: A metric optimized for short strings like personal names, giving higher scores to strings with matching prefixes.

Probabilistic linkage does not force a binary decision on all pairs. It defines a grey area between clear matches and clear non-matches where the statistical evidence is ambiguous. These borderline pairs are flagged for clerical review, a manual human adjudication process.

• Upper Threshold: Pairs scoring above this are automatically classified as matches.
• Lower Threshold: Pairs scoring below this are automatically classified as non-matches.
• Human-in-the-Loop: The clerical review ensures high-quality resolution for the most difficult edge cases, preventing both false positives and false negatives.

A critical advantage over deterministic linkage is the ability to make informed decisions even when key fields are blank. The model calculates a partial agreement weight that reflects the probability of a field being missing, rather than treating the absence of data as a disagreement.

• No Penalty for Nulls: A missing value contributes a neutral or slightly adjusted weight, avoiding the negative penalty that a direct mismatch would incur.
• Incomplete Evidence: The system can still link records confidently if the available fields provide sufficient statistical evidence, maximizing match recall in sparse datasets.

The discriminatory power of an identifier is directly tied to its rarity. Probabilistic linkage automatically adjusts weights based on the frequency of a value in the population. An agreement on a rare surname like 'Zzyzx' provides much stronger evidence for a match than an agreement on a common surname like 'Smith'.

• Inverse Frequency Weighting: The u-probability (random agreement) is estimated from the data distribution.
• Dynamic Calibration: This ensures that common values do not artificially inflate match scores, reducing false positives.

After pairwise scoring, the results are often represented as a graph where nodes are records and edges are potential matches. Transitive closure is applied to merge all connected components into a single entity cluster. If record A matches B, and B matches C, then A, B, and C are all resolved into a single golden record, ensuring linkage consistency across the entire dataset.

The foundational statistical framework for probabilistic record linkage. It computes match weights by calculating the log-likelihood ratio of two conditional probabilities: the m-probability (chance a field agrees given a true match) and the u-probability (chance a field agrees by random coincidence).

• Agreement weight: log2(m/u)
• Disagreement weight: log2((1-m)/(1-u))
• Composite weight sums across all fields to classify pairs as match, non-match, or clerical review.

A string similarity metric optimized for short strings like personal names. It calculates the proportion of matching characters within a defined window, then applies a prefix bonus to favor strings that match from the start.

• Score range: 0 (no similarity) to 1 (exact match)
• Prefix scale factor typically set to 0.1
• Handles transpositions more gracefully than Levenshtein distance
• Widely used for fuzzy name matching in probabilistic linkage engines.

The strategic process of partitioning datasets into mutually exclusive blocks to avoid the quadratic complexity of all-pairs comparison. A well-chosen blocking key groups records likely to match while keeping blocks small.

• Common keys: Phonetic encodings (Soundex), ZIP codes, birth year
• Reduces comparisons from n*m to sum(b_i²) where b_i is block size
• Multi-pass blocking uses multiple keys to recover missed matches across different blocking schemes.

The process of setting cutoff values on the composite similarity score to automatically classify record pairs. Thresholds define three zones:

• Upper threshold: Pairs above are auto-accepted as matches
• Lower threshold: Pairs below are auto-rejected as non-matches
• Clerical review zone: Pairs between thresholds require human adjudication

Optimal thresholds are determined by analyzing the trade-off between false match rate and false non-match rate against a ground-truth dataset.

The preprocessing step that transforms raw data into a consistent canonical format before linkage. Without standardization, even sophisticated probabilistic models fail due to irreconcilable format differences.

• Parsing: Splitting full names into given/family components
• Cleaning: Removing punctuation, whitespace normalization
• Normalization: Address standardization (e.g., 'St.' → 'Street')
• Phonetic encoding: Generating Soundex or Double Metaphone keys for fuzzy blocking

A graph-based resolution technique that ensures linkage consistency across an entire dataset. After pairwise comparisons, records are represented as nodes in a graph with edges indicating matches. Transitive closure identifies all connected components and merges them into a single entity cluster.

• If A matches B and B matches C, then A, B, and C form one entity
• Prevents fragmented clusters where the same real-world entity is split across multiple resolved records
• Essential for building a true golden record.