Identity Resolution

A rule-based approach that links records only when a specific set of identifiers match exactly. This method relies on a unique, high-quality key such as an email address, Social Security Number, or a composite key of multiple fields.

• Mechanism: Boolean logic (AND/OR rules) on exact field comparisons.
• Use Case: Ideal for systems with clean, standardized data and a reliable universal identifier.
• Limitation: Fails on typographical errors, nicknames, or missing data; generates high precision but low recall.

A statistical framework, formalized by the Fellegi-Sunter model, that calculates match weights for record pairs based on the agreement and disagreement patterns of multiple attributes. It estimates the probability that two records refer to the same entity without requiring exact matches.

• Key Metrics: Agreement weight (m-probability) and disagreement weight (u-probability).
• Advantage: Tolerates noise, typos, and missing data by using frequency-based likelihood ratios.
• Output: A composite score that can be thresholded to classify pairs as matches, non-matches, or potential matches for clerical review.

A string comparison technique that calculates the edit distance or similarity between two text strings to identify non-exact duplicates. It tolerates typographical errors, abbreviations, and formatting inconsistencies.

• Algorithms: Levenshtein distance, Jaro-Winkler, Soundex, and n-gram similarity.
• Application: Used to compare names ('Jon' vs 'John'), addresses ('St.' vs 'Street'), and other semi-structured text fields.
• Critical Tuning: Requires careful threshold setting to balance false positives against false negatives.

A computational efficiency technique that partitions datasets into mutually exclusive blocks using a cheap heuristic, drastically reducing the number of required pairwise comparisons from a Cartesian product to manageable subsets.

• Goal: Avoid O(n²) complexity by only comparing records within the same block.
• Common Keys: Phonetic encoding (Soundex), zip code, or first letter of the last name.
• Trade-off: Reduces computational load but risks missing true matches that fall across block boundaries (pair completeness vs. reduction ratio).

Uses supervised, unsupervised, or active learning models to classify record pairs as matches or non-matches. Modern approaches leverage transformer-based neural networks and entity embeddings to capture semantic similarity beyond surface-form string matching.

• Features: Vector similarity of name/address embeddings, temporal proximity, and relationship graph connectivity.
• Clustering: Algorithms like DBSCAN or hierarchical agglomerative clustering group all records belonging to the same entity.
• Advantage: Adapts to domain-specific patterns and improves over time with human-in-the-loop feedback.

A set of cryptographic and encoding techniques that allows the linkage of records across databases without revealing the plaintext personally identifiable information (PII) to any party.

• Techniques: Bloom filter encoding, homomorphic encryption, and secure multi-party computation.
• Use Case: Critical for inter-hospital patient matching or cross-government fraud detection where data cannot be shared in the clear.
• Challenge: Balancing linkage accuracy against the information loss introduced by the privacy-preserving encoding.

The statistical process of identifying and joining records across different datasets that correspond to the same entity when a reliable unique identifier is absent. This is the foundational mathematical backbone of identity resolution.

• Deterministic Matching: Exact agreement on a unique key (e.g., Social Security Number)
• Probabilistic Matching: Uses the Fellegi-Sunter model to calculate match weights based on attribute agreement/disagreement patterns
• Key Challenge: Balancing precision and recall when data quality is poor or schemas are heterogeneous

A computational efficiency technique that partitions datasets into mutually exclusive blocks using a cheap heuristic, drastically reducing the required number of pairwise comparisons from quadratic to near-linear.

• Standard Blocking: Groups records by exact match on a blocking key (e.g., ZIP code, first letter of surname)
• Sorted Neighborhood: Slides a window over sorted records to catch minor variations
• Canopy Clustering: Uses cheap distance metrics to create overlapping blocks, preventing false negatives from near-boundary records

A string comparison technique that calculates the similarity between two text strings to identify non-exact duplicates, tolerating typographical errors, abbreviations, and formatting inconsistencies.

• Levenshtein Distance: Minimum number of single-character edits required to transform one string into another
• Jaro-Winkler: Optimized for short strings like names, giving higher scores to strings that match from the beginning
• Phonetic Algorithms: Soundex and Metaphone index words by their pronunciation to catch spelling variants

The enterprise discipline of creating a single, trusted, authoritative golden record for critical business entities like customers and products by consolidating data from multiple source systems.

• Registry Style: Maintains a central index of cross-referenced keys without moving source data
• Consolidation Style: Physically merges records into a central hub for downstream consumption
• Coexistence Style: Allows bidirectional synchronization between the golden record and source systems

A statistical framework, formalized by the Fellegi-Sunter model, that calculates match weights for record pairs based on the agreement and disagreement patterns of multiple attributes without requiring a perfect unique key.

• M-Probability: The likelihood that a matching attribute agrees in a true match
• U-Probability: The likelihood of random agreement by chance
• Composite Weight: Sum of log-likelihood ratios across all fields, with a threshold decision boundary separating matches, non-matches, and a clerical review zone

A set of techniques that allows the linkage of records across databases without revealing the plaintext personally identifiable information to any party, critical for healthcare and cross-jurisdictional use cases.

• Bloom Filter Encoding: Converts identifiers into bit arrays using multiple hash functions, enabling approximate matching on encrypted tokens
• Phonetic Encoding: Hashes phonetic representations to preserve fuzzy matching capability in the encrypted space
• Secure Multi-Party Computation: Distributes computation so no single party sees the other's raw data