User Entity Resolution

The two fundamental approaches to linking records. Deterministic matching uses a unique, persistent identifier (e.g., hashed email, phone number) to link records with 100% certainty. Probabilistic matching uses statistical models (e.g., Fellegi-Sunter) to calculate a likelihood score based on non-unique attributes like IP address, device type, and browser fingerprint.

• Deterministic: High precision, low recall. Fails when identifiers change.
• Probabilistic: High recall, requires careful threshold tuning to manage false positives.
• Hybrid: Production systems typically combine both, using deterministic matches as anchors for probabilistic clustering.

The core data structure that stores resolved identities. An identity graph is a connected network of nodes (identifiers) and edges (match relationships). When two identifiers are matched, their subgraphs merge into a single cluster representing one user.

• Persistent ID: A unique, system-generated user_id is assigned to each cluster.
• Temporal Awareness: Edges must carry timestamps to handle identifier reuse (e.g., a phone number reassigned to a new person).
• Graph Databases: Often powered by tools like Apache TinkerPop or Neo4j for efficient traversal and merging of complex, multi-hop relationships.

The logic that governs how records are compared. Match rules define which attributes to compare, the comparison functions (e.g., exact match, Levenshtein distance, phonetic encoding like Soundex), and the weight assigned to each.

• Blocking Keys: A performance optimization that groups similar records (e.g., by ZIP code) to avoid an O(n²) comparison of all records.
• Thresholds: A match score above an upper threshold creates an auto-merge; a score below a lower threshold creates a separate entity. Scores in between are routed to a clerical review queue for human adjudication.

Once records are linked, the system must merge conflicting attribute values into a single 'best' version of the truth—the golden record. Survivorship rules dictate which source system's value wins for each attribute.

• Recency: The most recently updated value survives.
• Source Trust: A value from a CRM system may override one from a web form.
• Frequency: The most commonly occurring value across all linked records wins.
• Lineage: The golden record must retain an audit trail back to the source records that contributed to it for compliance and debugging.

The latency profile for identity resolution depends on the use case. Batch processing (e.g., nightly Hadoop/Spark jobs) is suitable for analytics and CRM hygiene. Real-time resolution is critical for in-session personalization and fraud detection.

• Real-time APIs: A low-latency service that, on receiving a new event, instantly queries the identity graph to return a resolved user_id.
• Lambda Architecture: A common pattern combining a speed layer (real-time stream processing with Apache Kafka/Flink) for immediate resolution and a batch layer for periodic graph recomputation and correction of streaming errors.

Entity resolution inherently creates a more complete—and therefore more sensitive—user profile. Architectures must embed privacy by design to comply with GDPR and CCPA.

• Right to be Forgotten: The identity graph must support the hard deletion or dissociation of all identifiers for a user upon request, a complex operation in a merged cluster.
• Data Minimization: Only attributes essential for matching should be ingested and stored.
• Pseudonymization: The persistent user_id should be a non-reversible token, decoupled from the raw identifiers used for matching, stored in a separate, access-controlled vault.