Entity Resolution

Entity resolution engines operate on a spectrum between rigid rule-based logic and statistical inference.

• Deterministic Matching: Requires exact agreement on a predefined combination of identifiers (e.g., SSN + DOB). It is fast and auditable but fails against typos or intentional obfuscation.
• Probabilistic Matching: Uses the Fellegi-Sunter model to calculate match likelihoods based on attribute agreement and disagreement patterns, tolerating real-world data noise.
• Hybrid Approaches: Modern production systems often cascade deterministic blocking keys with probabilistic scoring to balance speed and recall.

Comparing every record against every other record is computationally intractable (O(n²)). Blocking partitions datasets into mutually exclusive buckets to drastically reduce comparisons.

• Blocking Keys: Derived attributes like Soundex(LastName) + ZIP_Code group likely matches into the same block.
• Sorted Neighborhood: A sliding window passes over sorted records, comparing only those within a fixed distance.
• Canopy Clustering: Uses cheap, high-recall similarity metrics to create overlapping clusters before applying expensive, high-precision comparisons.
• Locality-Sensitive Hashing (LSH): Hashes high-dimensional vectors (like TF-IDF embeddings) so similar items collide in the same bucket with high probability.

The accuracy of entity resolution depends on selecting the right distance functions for each attribute type.

• String Fields: Levenshtein Distance for general typos; Jaro-Winkler for personal names where prefix matching matters; Soundex or Metaphone for phonetic normalization.
• Numeric Fields: Absolute difference or percentage tolerance for age; exact match for immutable identifiers.
• Text Fields: TF-IDF Vectorization followed by Cosine Similarity to compare addresses or business descriptions as semantic vectors.
• Temporal Fields: Date proximity scoring with configurable decay windows.
• Composite Scoring: Individual attribute similarities are aggregated into a weighted sum, with weights often learned via machine learning rather than manually assigned.

Pairwise matching alone is insufficient; records must be clustered into connected components representing distinct identities.

• Identity Clustering: Transforms pairwise match decisions into a graph where nodes are records and edges represent a match above threshold.
• Connected Components: The simplest clustering method—any path between two nodes merges them into the same entity.
• Community Detection: Algorithms like Louvain or Label Propagation identify tightly connected subgraphs, useful for detecting synthetic identity rings.
• Link Prediction: Graph neural networks predict missing edges, inferring hidden relationships between seemingly unconnected records—critical for uncovering fraud rings using stolen identity fragments.

When matching records across organizational boundaries—such as between banks for AML compliance—plaintext PII cannot be shared. Cryptographic protocols enable resolution without exposure.

• Bloom Filter Encoding: Sensitive attributes are hashed into irreversible bit arrays, allowing fuzzy matching via set overlap metrics like Dice coefficient without revealing original values.
• Homomorphic Encryption: Allows computation on encrypted data, producing encrypted match results that only the data owner can decrypt.
• Secure Multi-Party Computation (SMPC): Multiple parties jointly compute a matching function without revealing their private inputs to each other.
• Differential Privacy: Adds calibrated noise to match results to prevent inference attacks on individual records.

Entity resolution is not a one-time batch operation. Production systems must handle continuous data ingestion and evolving identities.

• Incremental Deduplication: New records are compared against existing resolved entities without recomputing the entire graph.
• Merge and Unmerge Logic: Systems must support rollback when a previously merged entity is later determined to be distinct due to new evidence.
• Temporal Decay: Match confidence decays over time for attributes that change (addresses, phone numbers), preventing stale links.
• Stream Processing: Real-time fraud systems perform entity resolution on event streams using sliding windows and in-memory state stores, resolving identities in milliseconds before transaction authorization.

A string comparison technique that calculates the probability of two text strings being a match, tolerating typographical errors, abbreviations, and formatting inconsistencies. Essential for resolving identities where names like 'Robert Smith' and 'Bob Smyth' may refer to the same individual.

• Levenshtein Distance: Measures minimum single-character edits required to transform one string into another
• Jaro-Winkler Similarity: Optimized for short strings like personal names, giving higher weight to matching prefixes
• Soundex: Phonetic algorithm indexing names by pronunciation, converting 'Smith' and 'Smyth' to the same code

These algorithms are the workhorses of entity resolution pipelines, applied to names, addresses, and organizational identifiers.

A statistical framework formalized by the Fellegi-Sunter model that uses likelihood ratios to calculate match probabilities between record pairs without relying on deterministic keys. This approach is critical when identifiers are noisy or deliberately obfuscated.

• Agreement weights: Quantify the evidentiary value of matching fields
• Disagreement weights: Penalize mismatches proportionally to field reliability
• Threshold optimization: Balances false positives against false negatives using Expectation-Maximization algorithms

The model estimates the probability that two records belong to the same entity given observed agreement patterns across multiple attributes, enabling robust identity resolution even with intentionally corrupted data.

An approach that models records as nodes and their pairwise similarity scores as edges in a graph, applying community detection or clustering algorithms to resolve entities. This method excels at uncovering synthetic identity rings where fraudsters create multiple interconnected fake profiles.

• Connected components: Identifies clusters of records all linked by high similarity scores
• Community detection: Algorithms like Louvain or Label Propagation partition the graph into distinct identity clusters
• Link prediction: Estimates the probability of hidden connections between seemingly unrelated identities

Graph-based methods reveal the structural patterns of fraud rings that pairwise comparison alone would miss, exposing the coordinated nature of synthetic identity attacks.

A performance optimization technique that partitions large datasets into mutually exclusive blocks using a common attribute, drastically reducing the number of record pair comparisons required. Without blocking, comparing 100 million records would require approximately 5 quadrillion pairwise comparisons.

• Standard blocking: Groups records by exact match on a key like ZIP code or birth year
• Sorted neighborhood: Sorts records by a blocking key and compares only within a sliding window
• Canopy clustering: Uses cheap, approximate distance metrics to create overlapping blocks

Effective blocking strategies reduce computational complexity from O(n²) to near-linear time while maintaining high recall, making real-time entity resolution feasible at scale.

A cryptographic protocol that enables the matching of records across disparate databases without revealing the plaintext personally identifiable information (PII) to any party. This is essential when financial institutions collaborate to detect synthetic identities spanning multiple organizations.

• Bloom filter encoding: Converts sensitive attributes into irreversible bit arrays that support fuzzy matching
• Homomorphic encryption: Allows computation on encrypted data without decryption
• Secure multi-party computation: Distributes computation so no single party sees the full dataset

These techniques enable consortium-based fraud detection while maintaining compliance with regulations like GDPR and CCPA, allowing banks to share identity intelligence without exposing customer data.