Identity Resolution

Deterministic matching uses exact or rule-based logic to link records. It relies on unique, stable identifiers or a combination of attributes that, when they match, provide definitive proof of entity sameness.

• Primary Method: Uses keys like government IDs (SSN), email addresses, or customer IDs.
• Rule-Based Logic: Employs business rules, e.g., IF (First_Name, Last_Name, Date_of_Birth, Postal_Code) match THEN link.
• Characteristics: Provides high precision but lower recall, as it cannot link records with missing or inconsistent data. It is the core of entity linking in master data management.

Probabilistic matching uses statistical models to calculate the likelihood that two records refer to the same entity. It is essential for handling noisy, incomplete, or unstructured data where exact matches are rare.

• Primary Method: Employs algorithms like Fellegi-Sunter to assign match, non-match, or potential-match weights to attribute comparisons.
• Fuzzy Matching: Incorporates techniques like Levenshtein distance for names or phonetic algorithms (Soundex, Metaphone) to account for typos and variations.
• Characteristics: Increases recall by finding non-obvious links but requires careful threshold tuning to balance precision and recall.

Graph-based resolution models entities and their attributes as nodes in a property graph, using the network structure itself to infer identity. Relationships and shared connections provide powerful contextual signals.

• Primary Method: Uses algorithms like node similarity, label propagation, or community detection to cluster records likely representing the same entity.
• Leverages Relationships: Infers identity from shared addresses, devices, transaction partners, or other associative links.
• Characteristics: Excels at uncovering complex, indirect relationships and is integral to building a knowledge graph where identity is a first-class construct.

Advanced machine learning models, particularly deep learning, learn a unified representation (embedding) of entity records to perform resolution in a high-dimensional semantic space.

• Primary Method: Models like Siamese Networks or Transformer encoders are trained to produce embeddings where records of the same entity are close together.
• Handles Multi-Modal Data: Can jointly process text, images, and behavioral signals to create a composite entity profile.
• Characteristics: Provides state-of-the-art accuracy for complex, large-scale resolution but requires significant labeled training data and computational resources.

After matching records, canonicalization creates a single, authoritative 'golden record' for each resolved entity. Record fusion is the process of merging attribute values from all source records into this canonical form.

• Primary Method: Applies conflict resolution rules (e.g., 'most recent', 'most frequent', 'highest confidence source') to select or synthesize the best value for each attribute.
• Creates Master Data: The output is a clean, unified profile used to populate the knowledge graph or customer data platform (CDP).
• Characteristics: Critical for data quality and ensuring downstream systems consume a consistent, trusted view of each entity.

An identity graph is a dynamic, persistent database that maintains all known identifiers, attributes, and linkages for an entity across systems and over time. It is the operational output of resolution.

• Primary Method: Assigns a persistent, system-generated universal unique identifier (UUID) to each resolved entity, which all applications use as the primary key.
• Manages Linkages: Stores the provenance and confidence of each link, enabling data lineage and auditability.
• Characteristics: Serves as the central source of truth for entity identity, powering real-time applications like personalization and fraud detection. It is the backbone of a semantic data fabric.

Fuzzy matching is an approximate string matching technique used to identify non-identical text entries that likely refer to the same real-world entity. It is essential for identity resolution when source data contains typographical errors, abbreviations, or formatting inconsistencies.

• Algorithms: Common methods include Levenshtein distance (edit distance), Jaro-Winkler similarity, and phonetic algorithms like Soundex.
• Use Case: Matching "Jon Smiht Corp." from one source with "John Smith Corporation" from another.
• Implementation: Often used in conjunction with deterministic rules and machine learning classifiers to improve match confidence.

Canonicalization is the process of converting data that has more than one possible representation into a single, standard, authoritative form (the canonical form). It is a critical transformation step that precedes or is integral to identity resolution.

• Objective: To ensure consistency for comparison and linking.
• Examples:
  • Converting "St.", "Street", and "St" to a single standard "Street".
  • Normalizing phone numbers to an international E.164 format (e.g., +1-555-123-4567).
  • Standardizing date formats to ISO 8601 (YYYY-MM-DD).
• Result: Reduces the complexity of matching logic by eliminating superficial variance.

Deduplication is the process of identifying and removing duplicate records within a single dataset that refer to the same real-world entity. It is a foundational step before cross-source identity resolution can occur.

• Scope: Intra-dataset cleansing.
• Techniques: Uses blocking (grouping likely matches, e.g., by postal code) and pairwise comparison using matching algorithms.
• Business Impact: Eliminates redundant customer profiles, prevents double-counting in analytics, and reduces storage costs.
• Key Challenge: Distinguishing between true duplicates (same entity) and distinct entities with similar attributes.

Schema alignment is the process of establishing semantic correspondences between the attributes, tables, or classes of two or more heterogeneous data schemas to enable integration. It provides the structural map for identity resolution rules.

• Input: Source and target schemas (e.g., a CRM Customer table and an ERP Client table).
• Output: A mapping stating that Customer.email is equivalent to Client.contact_email, and Customer.company_name is a sub-property of Client.org_name.
• Methods: Can be manual, rule-based, or automated using machine learning to infer correspondences.
• Role: Defines which fields should be compared during the identity resolution process.