Entity Resolution

The computational strategy used to avoid the O(n²) complexity of comparing every record to every other record. Blocking partitions the dataset into mutually exclusive buckets or blocks based on a blocking key (e.g., first 3 letters of last name, zip code). Only records within the same block are compared.

• Sorted Neighborhood: Sliding a window over sorted keys to cluster similar records.
• Canopy Clustering: Using a cheap, approximate distance metric to create overlapping subsets.
• Hashing: Applying Locality-Sensitive Hashing (LSH) to map similar records to the same hash bucket with high probability.

The primary goal is to maximize pairs completeness (recall of true matches) while drastically reducing the number of comparisons.

The core decision engine that determines if two records refer to the same entity. A match function computes a similarity score between 0.0 and 1.0 for each attribute pair. These attribute scores are aggregated into a composite match score using a decision model.

• Probabilistic Matching (Fellegi-Sunter): Calculates agreement and disagreement weights based on conditional probabilities (m-probability and u-probability) to produce a log-likelihood ratio.
• Deterministic Matching: Uses a cascade of exact-match rules (e.g., SSN matches AND DOB matches).
• Machine Learning Classifiers: Supervised models (XGBoost, Random Forest) trained on labeled pairs to classify them as match/non-match.
• Distance Metrics: Edit distance (Levenshtein), Jaro-Winkler, cosine similarity on embeddings, and phonetic algorithms (Soundex, Metaphone).

The process of converting pairwise match decisions into connected components of records that all represent the same entity. If record A matches B, and B matches C, then A, B, and C must belong to the same entity cluster.

• Connected Components: A graph algorithm that finds all disjoint subgraphs where nodes (records) are connected by edges (match decisions).
• Correlation Clustering: An optimization approach that minimizes disagreements (edges cut within a cluster, edges placed between clusters) to resolve inconsistencies.
• Hierarchical Agglomerative Clustering: Iteratively merges the most similar clusters based on a linkage criterion (single, complete, average).
• Canonicalization: Selecting the single best 'golden record' representation from the merged cluster attributes.

A critical pre-processing stage that transforms raw attributes into a consistent, comparable format. Without standardization, even identical entities will fail to match due to superficial string differences.

• Parsing: Decomposing a single string (e.g., 'John A. Smith Jr.') into atomic tokens (GivenName: John, MiddleName: A, FamilyName: Smith, Suffix: Jr.).
• Validation: Correcting values against a trusted reference table (e.g., USPS address validation).
• Transformation: Applying functions like UPPER(), SOUNDEX(), or date normalization (01-JAN-2024 -> 2024-01-01).
• Enhancement: Appending missing attributes from external APIs (e.g., geocoding an address to add latitude/longitude).

A modern approach that treats entity resolution as a graph problem from the start. Entities are nodes, and relationships (co-authorship, transactions, shared addresses) are edges. Resolution becomes a link prediction or node merging task.

• Knowledge Graph Embeddings: TransE, RotatE, or ComplEx models learn low-dimensional vector representations of nodes and edges to predict missing sameAs links.
• Graph Neural Networks (GNNs): Models like GraphSAGE or GAT aggregate features from a node's neighborhood to determine if two nodes should be merged.
• Community Detection: Algorithms like Louvain or Label Propagation can identify dense subgraphs that correspond to a single real-world entity.
• This method excels at resolving entities with sparse attributes but rich relational data.

The architectural pattern for resolving new records against an existing master data set without re-processing the entire database. This is essential for operational systems (CRM, CDP) where latency is critical.

• Insert-Only Matching: A new record is matched against the existing cluster centroids or canonical records, not every historical record.
• Split/Merge Management: Logic to handle the case where a new record causes an existing cluster to split into two distinct entities or two clusters to merge.
• Streaming Architectures: Using Apache Kafka or Flink to process a continuous stream of records, often employing Bloom filters for rapid blocking lookups.
• Delta Detection: Change Data Capture (CDC) mechanisms that identify only the modified attributes to trigger re-resolution.