Entity Resolution

A rule-based approach that classifies a record pair as a match only if two or more identifier attributes agree exactly. This method relies on unique, high-quality keys such as Social Security Numbers, email addresses, or composite business keys.

• Logic: Boolean AND/OR rules on exact field comparisons.
• Strength: High precision; zero false positives if the key is truly unique.
• Weakness: Brittle to typos, missing data, or format variations; a single character difference causes a miss.
• Example: IF (first_name == "John" AND last_name == "Smith" AND date_of_birth == "1990-01-01") THEN MATCH

Uses statistical models to calculate the likelihood that two records refer to the same entity, even when data contains errors, omissions, or inconsistencies. The Fellegi-Sunter model is the classic framework, assigning agreement and disagreement weights to each field.

• Mechanism: Computes a composite match score; pairs above a threshold are linked, those below are non-matches, and scores in between are sent for clerical review.
• Key Metric: Log-likelihood ratio comparing the probability of agreement given a match vs. a random chance.
• Example: "Jon Smith" with DOB 1/1/90 and "John Smyth" with DOB 1990-01-01 receive a high match score despite fuzzy differences.

A performance optimization technique that avoids the computationally prohibitive O(n²) comparison of every record against every other record. Records are partitioned into blocks based on a blocking key, and comparisons are only performed within each block.

• Blocking Key: A field or hash of fields with high selectivity, such as ZIP code, phonetic surname code, or n-gram index.
• Sorted Neighborhood: A sliding window is passed over a sorted dataset, comparing only records within the window.
• Canopy Clustering: Uses a cheap, approximate distance metric to create overlapping subsets before applying an expensive, precise comparison.

Treats entity resolution as a supervised binary classification problem. A model is trained on labeled pairs of records (match/non-match) to learn complex, non-linear decision boundaries that outperform manually tuned rules.

• Features: Field-level similarity scores (Jaccard, Levenshtein, TF-IDF cosine), numeric differences, and business-specific flags.
• Algorithms: Gradient-boosted trees (XGBoost) and neural networks excel at capturing feature interactions.
• Active Learning: An iterative process where the model identifies the most uncertain pairs for human labeling, dramatically reducing the manual annotation burden.

The final stage where pairwise match decisions are consolidated into connected components, each representing a single real-world entity. This process applies transitive closure: if A matches B, and B matches C, then A, B, and C all belong to the same canonical cluster.

• Connected Components: A graph algorithm that finds all records linked by a path of match edges.
• Correlation Clustering: A global optimization that minimizes disagreements between pairwise classifications and final cluster assignments, resolving inconsistencies where A matches B and B matches C, but A is classified as a non-match to C.
• Output: Each cluster is merged into a single golden record using survivorship rules.

Leverages deep neural networks to generate dense vector representations of entire records or entity descriptions, capturing semantic similarity beyond surface-level string overlap. This is particularly effective for unstructured text and multilingual data.

• Architecture: Siamese networks or pre-trained transformers (e.g., BERT) encode records into a shared vector space where matching entities are close together.
• Similarity: Cosine similarity between embeddings replaces traditional field-by-field fuzzy matching.
• Advantage: Handles paraphrasing, abbreviations, and cross-lingual variations where "United Nations" and "UN" have zero character overlap but identical meaning.

The single, best-curated version of a data entity that serves as the authoritative, canonical source of truth after a deduplication and merge process. Golden records are the ultimate output of entity resolution pipelines.

• Constructed by applying survivorship rules to conflicting values
• Represents the consolidated view of a customer, product, or organization
• Example: Merging 'Bob Smith', 'Robert Smith', and 'R. Smith' into one master profile

A data matching technique that identifies non-identical but probabilistically similar text strings, used to link records containing typographical errors, abbreviations, or transliterations.

• Essential for resolving real-world data quality issues
• Common algorithms include Levenshtein Distance and Jaro-Winkler
• Example: Matching 'Jon Stewart' with 'John Stuart' across CRM and billing systems

A metric measuring the cosine of the angle between two non-zero vectors in a multi-dimensional space. In entity resolution, it quantifies the semantic similarity between text embeddings or TF-IDF vectorized documents.

• Values range from -1 (opposite) to 1 (identical)
• Used for near-duplicate detection and semantic matching
• Example: Identifying that 'CEO' and 'Chief Executive Officer' refer to the same role

The logical inference that if record A matches record B, and record B matches record C, then A and C must also refer to the same entity, forming a complete canonical cluster.

• Critical for scalable entity clustering across millions of records
• Prevents fragmented identity groups
• Example: Linking three partial customer profiles from separate acquisitions into one unified identity

The process of linking disparate identifiers—such as cookies, device IDs, email addresses, and offline transactions—to create a unified, persistent canonical profile of an individual user across multiple touchpoints.

• Foundational for customer data platforms (CDPs)
• Uses deterministic and probabilistic matching rules
• Example: Connecting a user's mobile app activity with their desktop browsing and in-store purchases

An OWL property used in linked data and knowledge graphs to assert that two different URIs refer to the exact same real-world entity, facilitating identity reconciliation across datasets.

• Core mechanism of the Semantic Web and Wikidata
• Enables cross-database entity disambiguation
• Example: Asserting that dbpedia:New_York_City is owl:sameAs wikidata:Q60