Blocking Keys

A blocking key defines a deterministic function that assigns each record to exactly one block based on a specific attribute or combination of attributes. Records with identical blocking key values are placed in the same block, while records with different values are placed in separate blocks. This ensures that comparisons are only performed within blocks, never across them.

• Mutual Exclusivity: Each record belongs to one and only one block
• Common attributes: Zip code, birth year, or first letter of surname
• Hash-based keys: MD5 or SHA256 of concatenated fields for uniform distribution

The granularity of a blocking key directly impacts both recall and computational cost. A key that is too coarse (e.g., gender) creates oversized blocks that defeat the purpose of blocking. A key that is too fine (e.g., full phone number) risks missing true matches due to typos or missing data.

• High granularity: Small blocks, faster execution, lower recall
• Low granularity: Large blocks, slower execution, higher recall
• Optimal balance: Phonetic encodings like Soundex or Double Metaphone of surnames

To mitigate the recall loss from a single blocking key, multi-pass blocking executes multiple independent blocking runs using different keys and unions the resulting candidate pairs. This ensures that if a true match is missed by one key (e.g., due to a typo in the surname), it may still be captured by another key (e.g., date of birth + zip code).

• Union of candidates: Combines pairs from all passes, deduplicating before scoring
• Common passes: Surname + DOB, ZIP + first name, phone hash
• Transitive closure: If A matches B in pass 1 and B matches C in pass 2, A and C become candidates

An extension of traditional blocking where records are sorted by the blocking key and a fixed-size sliding window is moved over the sorted list. Only records within the same window are compared, allowing for fuzzy matching on the key itself. This is particularly effective when the blocking key may contain minor errors.

• Window size: Typically 10-50 records, balancing recall and speed
• Key sorting: Surname, company name, or geohash for spatial data
• Multi-pass variant: Sort on different keys and union results

Locality-Sensitive Hashing (LSH) is a probabilistic blocking technique that hashes similar input items into the same buckets with high probability. Unlike deterministic keys, LSH uses multiple hash functions to create signature bands, ensuring that records with high Jaccard or cosine similarity collide in at least one band.

• MinHash: LSH family for Jaccard similarity on sets (e.g., n-grams of names)
• SimHash: LSH family for cosine similarity on vectors (e.g., TF-IDF embeddings)
• Band size tuning: Smaller bands increase recall; larger bands increase precision

A lightweight, approximate clustering algorithm used as a pre-blocking step to create overlapping blocks (canopies) using a cheap, high-recall similarity metric. Records are first assigned to canopies using a loose threshold, and then expensive, precise comparisons are only performed within each canopy.

• T1 threshold: Loose bound for canopy creation, maximizing recall
• T2 threshold: Tight bound for seeding new canopies
• Common metric: TF-IDF cosine similarity on name and address tokens

The computational process of identifying, linking, and merging disparate records that refer to the same real-world entity across different data sources. Blocking keys serve as the indexing mechanism that makes entity resolution computationally feasible on large datasets by partitioning records into mutually exclusive blocks before pairwise comparison.

The statistical methodology for joining records from two or more datasets that are believed to correspond to the same entity when a common unique identifier is absent. Blocking keys are essential in record linkage to reduce the comparison space. Common blocking key strategies include:

• Phonetic encoding (Soundex, Metaphone) for name fields
• N-gram indexing for fuzzy text matching
• Sorted neighborhood methods using sliding windows over sorted keys

A string comparison technique that calculates the probability of two text strings being a match, tolerating typographical errors, abbreviations, and formatting inconsistencies. Blocking keys must be designed to accommodate fuzzy matching by using lossy transformations—such as Soundex or Double Metaphone—that map similar strings to the same block, ensuring that near-matches are not excluded from comparison.

A statistical framework, formalized by the Fellegi-Sunter model, that uses likelihood ratios to calculate match probabilities between record pairs without deterministic keys. Blocking keys in probabilistic systems are often generated from multiple attributes and combined with union and intersection strategies to balance recall and precision. The Fellegi-Sunter model classifies record pairs as matches, non-matches, or potential matches based on agreement pattern likelihood ratios.

A cryptographic protocol that enables the matching of records across disparate databases without revealing plaintext personally identifiable information to any party. Blocking keys in PPRL are typically encoded using Bloom filter encoding or homomorphic encryption, allowing organizations to collaboratively detect synthetic identities across institutional boundaries while maintaining regulatory compliance.

An unsupervised machine learning technique that groups multiple records or data points into clusters representing distinct real-world identities based on similarity metrics. After blocking keys partition the dataset, identity clustering algorithms—such as hierarchical agglomerative clustering or connected components on similarity graphs—operate within each block to resolve entities without requiring labeled training data.