Entity Resolution

Entity resolution systems employ two fundamental matching paradigms. Deterministic matching uses exact or rule-based comparisons (e.g., SSN == SSN) to declare a match with absolute certainty. Probabilistic matching calculates a likelihood ratio that two records represent the same entity based on the agreement and disagreement of multiple attributes, using algorithms like the Fellegi-Sunter model. Modern systems often combine both: deterministic rules for high-precision blocking and probabilistic scoring for final classification.

To avoid the computationally prohibitive O(n²) comparison of all record pairs, entity resolution relies on blocking. This step partitions the dataset into mutually exclusive blocks using blocking keys—derived from attributes like ZIP code or phonetic name encodings (e.g., Soundex, Double Metaphone). Only records within the same block are compared. Advanced techniques include sorted neighborhood (sliding a window over sorted data) and canopy clustering (using cheap, approximate metrics to create overlapping clusters).

The core of record comparison relies on attribute-level similarity functions. Common functions include:

• Levenshtein distance: Edit distance for typographical errors.
• Jaro-Winkler: Optimized for short strings like personal names.
• Jaccard index: Set-based overlap for multi-value attributes.
• TF-IDF cosine similarity: Weighted token importance for longer text fields. Feature engineering transforms raw attribute similarities into a feature vector for a machine learning classifier, often incorporating numeric distance, date proximity, and domain-specific rules.

Pairwise match decisions must be resolved into global clusters of records representing a single entity. Transitive closure is applied: if record A matches B, and B matches C, then A, B, and C form a cluster. However, real-world noise requires robust clustering algorithms like Hierarchical Agglomerative Clustering or Connected Components on a match graph. After clustering, canonicalization selects or synthesizes the single 'golden record'—the best composite representation of the entity—by applying survivorship rules (e.g., 'prefer the most recent address').

Supervised entity resolution models require labeled pairs of matching and non-matching records. Since manual labeling is expensive, active learning is critical. The system iteratively selects the most informative unlabeled pairs for human review, often using uncertainty sampling (pairs with a match probability near 0.5). Distant supervision can bootstrap training data by heuristically aligning records across sources using strong identifiers, though this introduces noisy labels that must be managed with robust loss functions.

Modern approaches model the entire dataset as a heterogeneous graph where nodes are records and edges represent weighted similarity relationships. Graph neural networks (GNNs) can learn to propagate match signals across the graph, leveraging structural information. Algorithms like Personalized PageRank can be used for collective resolution, where the decision to merge two records is influenced by the similarity of their respective neighbors. This is particularly effective for disambiguating entities in social networks or bibliographic databases.