Entity Resolution

The foundational logic for comparing records falls into two categories:

• Deterministic Matching: Relies on exact or rule-based agreement on a unique identifier (e.g., CUSIP, LEI). It is highly precise but fragile against data entry errors.
• Probabilistic Matching: Uses statistical models (like Fellegi-Sunter) to calculate a match probability based on the agreement and disagreement weights of multiple fields (e.g., name, address, date of birth). This handles noise and typos effectively.
• Hybrid Approaches: Production systems often cascade deterministic rules for high-confidence blocking before applying computationally expensive probabilistic models.

Comparing every record against every other record is an O(n²) computational impossibility at scale. Blocking partitions the dataset into mutually exclusive buckets (blocks) using a blocking key (e.g., first 3 letters of name + ZIP code). Only records within the same block are compared.

• Sorted Neighborhood Method: Sorts records by a key and slides a window to compare nearby records.
• Canopy Clustering: Uses cheap, approximate distance metrics to create overlapping clusters for comparison.
• Locality-Sensitive Hashing (LSH): Hashes similar input items into the same buckets with high probability, enabling efficient approximate nearest neighbor search on high-dimensional text embeddings.

Raw text must be transformed into comparison-ready features. Key techniques include:

• Phonetic Encoding: Algorithms like Soundex, Metaphone, and Double Metaphone index names by their pronunciation to catch spelling variations (e.g., 'Smith' vs. 'Smyth').
• String Metrics: Levenshtein distance (edit distance), Jaro-Winkler (optimized for name matching), and Cosine Similarity on TF-IDF vectors quantify textual similarity.
• Tokenization and Standardization: Address parsing, acronym normalization ('Intl' to 'International'), and unit conversion ensure semantic equivalence before comparison.

Pairwise match decisions must be resolved into connected components of records representing a single entity. This is a graph clustering problem.

• Transitive Closure: If record A matches B, and B matches C, then A, B, and C form a single cluster.
• Connected Components: An efficient graph algorithm to find all disjoint subgraphs in the match graph.
• Survivorship and Canonical Record Creation: Once a cluster is formed, a 'golden record' must be synthesized by selecting the most complete, recent, and trustworthy attribute values from across the cluster members using configurable survivorship rules.

Entity resolution is the algorithmic engine within a broader Master Data Management strategy. The MDM system governs the lifecycle of the golden record.

• Registry Style: Links records but leaves source data untouched, providing a unified view.
• Consolidation Style: Physically merges and stores the best version of the master data in a central hub.
• Coexistence Style: A hybrid where the golden record is stored centrally, and updates are synchronized back to source systems. The chosen style dictates the latency and authority of the resolved entities.

Model efficacy is measured by precision and recall, balanced by a match threshold.

• Precision: Of all pairs predicted as a match, what fraction are true matches? A low precision leads to false merges, which are costly to undo.
• Recall: Of all true matches, what fraction did the model find? Low recall misses fraud or risk exposure.
• Match Threshold Tuning: The probability score above which a pair is declared a match. Adjusting this threshold trades off precision against recall based on the business cost of a false positive vs. a false negative. A human review queue is often established for pairs in a 'probable' score band.

The critical preprocessing step that transforms heterogeneous raw data into a consistent, canonical format before matching. Without normalization, simple discrepancies like 'St.' vs 'Street' or 'IBM' vs 'International Business Machines' defeat exact-match algorithms.

• Standardization: Converting values to a common format (e.g., all-uppercase, ISO date formats)
• Parsing: Splitting free-text fields into atomic components (first name, last name, suffix)
• Validation: Verifying data against reference datasets (postal codes, ticker symbols)
• Transliteration: Converting characters between writing systems (e.g., 中文 to pinyin)

The end-to-end audit trail tracking how entity records are sourced, transformed, and merged across pipelines. In entity resolution, lineage is essential for debugging false positives and demonstrating regulatory compliance.

• Provenance tracking: Identifying the original source system for each merged attribute
• Transformation logging: Recording every cleaning, standardization, and matching step applied
• Impact analysis: Understanding which downstream models and reports are affected by a merge or split decision
• Reproducibility: Enabling the exact reconstruction of any entity cluster at a point in time

A historical snapshot preserving the exact state of an entity record as it was known on a specific past date. This is vital for entity resolution in backtesting, where using current corrected data would introduce look-ahead bias.

• Temporal consistency: Ensuring that entity linkages used in a 2018 simulation only use information available in 2018
• Slowly changing dimensions: Tracking how entity attributes (name, headquarters, ticker) evolve over time
• As-of queries: The ability to retrieve the resolved entity graph as it existed at any historical timestamp

A centralized platform for storing, versioning, and serving the features derived from entity resolution pipelines. Once entities are disambiguated, their resolved profiles become high-value features for downstream models.

• Consistent serving: Ensuring the same entity features are used in training and inference
• Point-in-time correctness: Joining entity features to training examples without temporal leakage
• Feature reuse: Allowing multiple trading models to consume the same resolved entity graph
• Versioning: Tracking how changes to resolution logic affect feature distributions over time

A design pattern for identifying and tracking incremental changes to source records, enabling low-latency entity resolution on streaming data. CDC captures inserts, updates, and deletes from transactional databases.

• Real-time resolution: Re-evaluating entity clusters as new transactions arrive
• Log-based capture: Reading database transaction logs for minimal source impact
• Event-driven architecture: Triggering re-resolution only for affected entities rather than full batch reprocessing
• Outbox pattern: Ensuring reliable propagation of entity change events to downstream systems

The ability to adapt entity resolution pipelines to handle changes in source data structure without breaking downstream consumers. As vendors add fields or change formats, the resolution logic must gracefully accommodate these shifts.

• Backward compatibility: Ensuring new schema versions can still read old data
• Forward compatibility: Designing parsers to ignore unknown future fields
• Schema registry: A central repository for managing and validating schema versions across the resolution pipeline
• Semantic mapping: Maintaining mappings when source field names change but meaning remains constant