Entity Resolution

Two foundational paradigms govern how records are linked. Deterministic matching relies on exact or rule-based agreement across a defined set of identifiers, such as a composite key of Social Security Number and date of birth. It produces binary outcomes: match or no-match. Probabilistic matching, by contrast, uses statistical models—often based on the Fellegi-Sunter framework—to calculate a likelihood ratio that two records represent the same entity despite typographical errors, missing fields, or temporal drift. This approach assigns agreement and disagreement weights to each attribute, enabling nuanced scoring in noisy real-world datasets.

A brute-force comparison of every record against every other record scales quadratically—O(n²)—and is computationally prohibitive for large datasets. Blocking solves this by partitioning the dataset into mutually exclusive buckets using a blocking key, such as the first three characters of a surname or a zip code prefix. Only records within the same block are compared. Sorted neighborhood indexing further refines this by sliding a window over sorted keys. The goal is to maximize the pairs completeness (recall of true matches) while minimizing the reduction ratio (the fraction of unnecessary comparisons pruned).

Entity resolution does not end with pairwise match scores; it requires grouping all records belonging to the same entity into a canonical cluster. This is a transitive closure problem: if record A matches B, and B matches C, then A, B, and C must be merged. Clustering algorithms like connected components or hierarchical agglomerative clustering build these groups. Once clustered, a canonical representation—often called a 'golden record'—must be synthesized by selecting the most accurate, complete, or recent attribute values from across the cluster members, a process known as survivorship.

The raw signals driving resolution are generated by comparing attribute values. Effective entity resolution depends on selecting the right similarity functions for each data type:

• String metrics: Levenshtein distance, Jaro-Winkler, and cosine similarity over character n-grams for names and addresses.
• Phonetic encoding: Soundex, Metaphone, and Double Metaphone to normalize pronunciation-based spelling variations.
• Numeric and temporal: Absolute difference, percentage tolerance, and date-fuzzy logic to handle transpositions and time-window logic.
• Embedding-based: Dense vector similarity from pre-trained NLP models to capture semantic equivalence of descriptions or product names.

A distinct but related task, entity linking resolves textual mentions in unstructured documents to a unique, canonical entry in a structured knowledge base like Wikidata or a proprietary master data management system. This involves three sub-steps: mention detection (identifying spans of text that refer to an entity), candidate generation (retrieving a shortlist of possible KB entries), and candidate disambiguation (ranking candidates using contextual similarity). This process transforms ambiguous strings like 'Paris' into the distinct entity Q90 (the capital of France) versus Q167 (the mythological figure).

In regulated industries and cross-organizational data collaborations, personally identifiable information cannot be shared in the clear. Privacy-Preserving Record Linkage (PPRL) techniques allow entity resolution on encrypted data. Common methods include:

• Bloom filter encoding: Attributes are hashed into bit arrays, allowing approximate matching via set-based similarity (Dice coefficient) without revealing plaintext.
• Homomorphic encryption: Computations are performed directly on ciphertexts, though this remains computationally expensive for large-scale linkage.
• Secure multi-party computation: Multiple parties jointly compute match results without revealing their private inputs to one another.

The target data structure for resolved entities. A knowledge graph stores canonical entities as nodes and their relationships as typed edges, forming a semantic network. Entity resolution populates these graphs by linking textual mentions to unique identifiers like Wikidata Q-IDs or internal master data management keys. This provides the deterministic factual grounding required for retrieval-augmented generation and reasoning systems.

The database-centric sibling of entity resolution. Record linkage identifies duplicate records that refer to the same real-world entity across structured datasets—such as customer databases with inconsistent spellings or addresses. Techniques include:

• Deterministic matching on exact keys
• Probabilistic matching using Fellegi-Sunter models
• Blocking to reduce the O(n²) comparison space

The linguistic task of identifying all expressions that refer to the same entity within a single document. This includes linking pronouns ('she', 'it'), definite descriptions ('the CEO'), and named mentions to a common referent. Entity resolution operates at a cross-document level, while coreference resolution handles within-document anaphora. Both are essential for building coherent entity-centric representations from text.

The mathematical foundation for modern entity resolution. Entities are encoded as dense vector representations in high-dimensional space, where semantically similar entities cluster together. Cosine similarity between embeddings enables fuzzy matching at scale. Techniques like dual-encoder architectures allow for efficient nearest-neighbor search across millions of candidate entities using approximate nearest neighbor (ANN) indexes.