Entity Resolution

A rule-based approach where records are linked if they agree exactly or within predefined tolerances on a set of identifier attributes. This method relies on exact matching or fuzzy matching on fields like Social Security Numbers or email addresses.

• Pros: Highly precise, fast, and easily auditable.
• Cons: Brittle; fails when data contains typos, transpositions, or missing values.
• Example: Linking customer records where first_name, last_name, and date_of_birth are identical.

Uses statistical models to estimate the likelihood that a pair of records refers to the same entity, even when attributes disagree. The Fellegi-Sunter model is the classic framework, calculating agreement and disagreement weights for each field.

• Key Concept: A field match on a rare value (e.g., a unique nickname) provides stronger evidence than a match on a common value (e.g., 'John').
• Output: A confidence score between 0 and 1, allowing for tiered review workflows.

A performance optimization technique that avoids the computationally prohibitive O(n²) comparison of all record pairs. Records are partitioned into blocks using a blocking key (e.g., Zip Code or Soundex of last name), and comparisons are restricted to records within the same block.

• Sorted Neighborhood Method: Slides a window over records sorted by a key, comparing only nearby records.
• Canopy Clustering: Uses a cheap, high-recall similarity measure to create overlapping clusters for subsequent detailed comparison.

Employs supervised, unsupervised, or active learning models to classify record pairs as matches or non-matches. Feature vectors are constructed from field-level similarity scores (e.g., Jaccard, Levenshtein, TF-IDF cosine distance).

• Supervised: Requires labeled training data of known duplicates and distinct records.
• Unsupervised: Uses clustering algorithms like DBSCAN on embedding vectors to group similar records without labeled examples.
• Active Learning: Iteratively queries a human oracle to label the most informative, uncertain pairs to efficiently train a model.

Represents records as nodes and their pairwise similarity scores as weighted edges in a graph. Entity resolution becomes a community detection or graph partitioning problem.

• Connected Components: A simple transitive closure—if A matches B, and B matches C, then A, B, and C are the same entity.
• Correlation Clustering: A more sophisticated global optimization that partitions the graph to minimize disagreements (non-match edges within a cluster, match edges across clusters), resolving inconsistencies in pairwise decisions.

Processes new or updated records without re-executing the entire batch resolution pipeline. This is critical for operational systems like Change Data Capture (CDC) feeds.

• Indexed Lookup: A new record is hashed and compared only against a small set of candidate golden records retrieved via a high-speed index.
• Delta Processing: Only blocks affected by the new or modified record are re-evaluated, maintaining the integrity of the master data without a full recomputation.

An information extraction subtask that locates and classifies named entities in unstructured text into predefined categories such as persons, organizations, and locations. NER is a critical pre-processing step for entity resolution on text corpora.

• Input: Raw, unstructured documents.
• Output: Tagged spans of text with labels like PERSON or ORG.
• Resolution Dependency: You cannot resolve an entity you haven't first identified. NER provides the candidate mentions for downstream resolution pipelines.

The NLP task of mapping ambiguous textual mentions of entities to their unique canonical identifiers in a knowledge base like Wikidata. While Entity Resolution merges records within your data, Entity Linking connects your data to the outside world.

• Disambiguation: Resolves 'Apple' to the company Q312 or the fruit Q89.
• KB Anchoring: Provides a universal, machine-readable ID for a real-world object.
• Synergy: A resolved internal golden record can be linked to a Wikidata ID for rich external context enrichment.

An algorithmic technique that hashes similar input items into the same buckets with high probability, enabling efficient approximate nearest neighbor search for deduplication. LSH is a core performance optimization for blocking in entity resolution.

• Blocking: Avoids the O(n²) problem of comparing every record to every other record.
• MinHash: A popular LSH family for estimating Jaccard similarity between text sets.
• Performance: Reduces comparisons from billions to millions by only matching records within the same hash bucket.

A comprehensive methodology for defining and managing an organization's critical data entities to provide a single, authoritative point of reference. Entity Resolution is the technical engine inside an MDM strategy.

• Governance: MDM adds the policies, stewardship, and workflows around the resolution process.
• Domains: Customer (CDM), Product (PDM), Supplier, etc.
• Implementation Styles: Registry, Consolidation, Coexistence, and Centralized—each using entity resolution differently.

A dimensionality reduction technique that maps nodes, edges, or entire subgraphs into a low-dimensional vector space while preserving the graph's topological structure. This enables similarity-based entity resolution using vector distance.

• Node2Vec/TransE: Algorithms that learn embeddings by predicting graph context or relationships.
• Vector Similarity: Resolved entities have embeddings that are close in vector space.
• Use Case: Identifying that node_A and node_B likely represent the same entity because their graph neighborhoods and attributes produce highly similar embedding vectors.