Entity Resolution

A rule-based approach that establishes record linkage through exact or partial agreement on predefined identifier fields. This method applies strict logical conditions—such as exact Social Security Number or composite key matches—to generate binary match/no-match outcomes.

• Exact Matching: Requires character-for-character equality on a unique identifier.
• Deterministic Rule Sets: Uses cascading logic, such as matching on first name, last name, and date of birth simultaneously.
• Guarantees: 100% precision for the defined rule but suffers from low recall due to typos or missing data.
• Use Case: Ideal for high-certainty linking in financial compliance where false positives are unacceptable.

A statistical framework, formalized by Fellegi and Sunter, that calculates the likelihood that two records belong to the same entity by comparing multiple fields and assigning agreement and disagreement weights. This method models the probability of chance agreement versus true match agreement.

• M-Probability: The probability that a field agrees given the pair is a true match.
• U-Probability: The probability that a field agrees by random chance.
• Composite Weight: Sum of log-likelihood ratios across all compared attributes.
• Threshold Tuning: A cutoff score is set to classify pairs as matches, non-matches, or potential matches requiring clerical review.

A performance optimization technique that reduces the quadratic computational complexity of comparing every record against every other record. Blocking keys partition the dataset into mutually exclusive buckets, ensuring only records within the same block are compared.

• Sorted Neighborhood Method: Slides a window over a sorted list of records to compare only nearby entries.
• Canopy Clustering: Uses a cheap, approximate distance metric to create overlapping subsets for candidate generation.
• Phonetic Blocking: Groups records by sound-based encodings like Soundex or Metaphone to catch phonetic spelling variations.
• Goal: Drastically reduce the number of pairwise comparisons from billions to millions.

Algorithms that quantify the lexical distance between two text strings to account for typographical errors, abbreviations, and formatting inconsistencies that defeat exact matching. These metrics are the core distance functions in probabilistic models.

• Levenshtein Distance: The minimum number of single-character edits (insertions, deletions, substitutions) required to change one string into another.
• Jaro-Winkler Similarity: A metric optimized for short strings like names, giving higher weight to matching prefixes.
• Cosine Similarity with TF-IDF: Treats strings as vectors of n-grams or tokens to measure angular similarity, robust to word transpositions.
• Phonetic Algorithms: Double Metaphone reduces names to a primary and secondary phonetic hash to handle diverse linguistic origins.

An approach that models entities and their relationships as a connected graph, where nodes represent records and edges represent match links. This enables transitive resolution, where if record A matches B, and B matches C, then A and C are inferred to be the same entity.

• Connected Components: Identifies clusters of records that are transitively linked, merging them into a single resolved entity.
• Community Detection: Algorithms like Louvain can identify dense clusters of identity in massive graphs.
• Edge Weighting: Match confidence scores are stored as edge weights, allowing for weighted graph traversal and pruning of weak links.
• Advantage: Resolves complex many-to-many relationships that pairwise matching misses.

The post-resolution process of constructing a single, authoritative golden record from a cluster of matched duplicate records. Survivorship rules are deterministic policies that select the best value for each attribute from the competing source records.

• Source Trustworthiness: A hierarchical ranking of data sources, where a value from a trusted source (e.g., passport) overrides a less trusted one (e.g., utility bill).
• Data Currency: Rules that prioritize the most recent timestamp or the most frequently occurring value.
• Attribute-Level Rules: Different survivorship strategies can apply to different fields; a name might take the longest value, while an address takes the most recent.
• Lineage Tracking: The golden record must retain pointers back to the source records that contributed to its values for auditability.