Entity Resolution

The foundational dichotomy in record linkage strategy. Deterministic matching relies on exact or rule-based agreement on a set of identifiers (e.g., SSN + DOB must match exactly). Probabilistic matching, rooted in the Fellegi-Sunter model, calculates the likelihood that two records refer to the same entity by weighing the agreement and disagreement of multiple fields.

• Deterministic: High precision, low recall. Fails on typos or missing data.
• Probabilistic: Handles noise and variation. Requires training data to estimate match weights.
• Hybrid approaches use deterministic rules for high-confidence blocking and probabilistic scoring for edge cases.

The computational strategy to avoid the O(n²) problem of comparing every record against every other. Blocking partitions the dataset into mutually exclusive buckets using a blocking key (e.g., ZIP code, Soundex of last name). Only records within the same block are compared.

• Sorted Neighborhood: Slides a window over sorted records to cluster potential matches.
• Canopy Clustering: Uses cheap, approximate distance measures to create overlapping clusters.
• Locality-Sensitive Hashing (LSH): Hashes similar items into the same buckets with high probability, critical for high-dimensional data.

The transformation of raw attribute values into comparison vectors and the application of distance functions. Common similarity metrics include:

• Levenshtein Distance: Edit distance for typographical errors.
• Jaro-Winkler: Optimized for short strings like names, giving higher weight to prefix matches.
• TF-IDF Cosine Similarity: Treats strings as token vectors, effective for addresses and descriptions.
• Phonetic Encoding (Soundex, Metaphone): Indexes strings by pronunciation to catch homophone errors.
• Numeric Delta: Absolute or percentage difference for dates, ages, and monetary values.

The final stage where pairwise match decisions are resolved into global entity clusters. A canonical entity (or survivor record) is selected to represent the cluster.

• Transitive Closure: If A matches B, and B matches C, then A, B, and C form a single cluster. Computationally intensive.
• Connected Components: A graph algorithm where records are nodes and match decisions are edges.
• Centroid Selection: The canonical record is chosen by data quality rules (e.g., most complete, most recent, most authoritative source).
• Persistent Identifiers: A unique, immutable ID is assigned to the cluster for downstream linking.

Modern approaches that train classifiers to predict match/non-match labels from labeled training data, replacing manually tuned rule weights.

• Supervised Classification: Random Forests, SVMs, or Gradient Boosting trained on feature vectors of record pairs.
• Active Learning: Iteratively queries a human oracle to label the most uncertain pairs, minimizing annotation cost.
• Deep Learning: Siamese neural networks or Transformer models (e.g., BERT for entity matching) that learn optimal representations directly from raw text, capturing semantic equivalence beyond string similarity.
• Unsupervised EM: The Expectation-Maximization algorithm estimates probabilistic model parameters without labeled data.

The specialized application of entity resolution to graph-structured data, often called entity alignment or instance matching. The goal is to identify nodes in different knowledge graphs (or different partitions of the same graph) that refer to the same real-world entity.

• Structural Similarity: Leverages graph topology—two nodes are likely the same if they share similar neighbors and relationships.
• Embedding-Based Alignment: Uses knowledge graph embeddings (e.g., TransE, RotatE) to align entities in a shared vector space.
• Property-Based Matching: Compares literal attributes (labels, dates, identifiers) using traditional string and numeric metrics.
• Owl:sameAs Linkage: The final output is often an owl:sameAs predicate connecting the two node URIs.