Blocking

The most fundamental blocking technique. Records are assigned to the same block if they share an identical value for a chosen blocking key, such as a ZIP code or the first three letters of a last name. It is deterministic and highly efficient but suffers from the curse of dimensionality—records with minor variations in the key are placed in different blocks and never compared.

• Example: Blocking all customer records on postal_code='90210'.
• Use Case: Initial, high-recall pass in a multi-pass strategy where speed is critical.

A technique that sorts the entire dataset based on a sorting key (e.g., concatenated last_name, first_name). A fixed-size sliding window (e.g., 10 records) moves down the sorted list, and only records within the same window are compared. This method is more robust to minor errors at the beginning of the key than standard blocking.

• Process: 1) Create sorting key, 2) Sort all records, 3) Slide window and compare within it.
• Advantage: Captures matches where standard blocking fails due to transpositions or typos early in the key.

A flexible method that uses overlapping substrings of length q (typically q=2 or 3, called bigrams or trigrams) as blocking keys. Each record generates multiple q-grams from an attribute, and records sharing any q-gram are placed in the same block. This dramatically increases recall for fuzzy matches.

• Example: The string 'Miller' generates bigrams: #M, Mi, il, ll, le, er, r#.
• Trade-off: Higher recall but produces larger, overlapping blocks, increasing comparison costs. Often used with prefix filtering to prune the number of q-grams considered.

An approximate, probabilistic blocking method designed for high-dimensional data like text embeddings. LSH uses special hash functions where the probability of collision is high for similar inputs and low for dissimilar ones. Similar records are hashed into the same bucket with high probability.

• Key Property: It is locality-sensitive.
• Common Use: Blocking for deep learning models where records are represented as dense vectors. Provides a sub-linear time alternative to comparing all pairs.
• Families: Includes MinHash for Jaccard similarity and SimHash for cosine similarity.

A fast, unsupervised pre-clustering technique used to create overlapping blocks (canopies). It uses two distance thresholds: a loose threshold T1 and a tight threshold T2 (T1 > T2). A record forms a new canopy if it is not within T2 of an existing canopy center. All records within T1 of a center are placed in that canopy.

• Result: Records can belong to multiple canopies.
• Purpose: Not a final clustering solution, but a cheap way to create candidate sets for more expensive pairwise matching or agglomerative clustering algorithms.

A robust production strategy that applies multiple, independent blocking schemes in sequence or parallel and takes the union of candidate pairs. This mitigates the risk of any single blocking key failing.

• Typical Workflow:
  1. Pass 1 (High Recall): Use a lenient key (e.g., Soundex of last name).
  2. Pass 2 (High Precision): Use a strict key (e.g., exact date of birth + first initial).
• Engineering Benefit: Allows tuning of the recall/precision trade-off systematically. The final candidate set is the union from all passes, ensuring no potential match is missed due to a single data error.

A family of algorithmic techniques used to create blocking keys. LSH functions hash input items so that similar items map to the same hash buckets (blocks) with high probability, while dissimilar items map to different buckets. This enables approximate nearest neighbor search at scale.

• Core Mechanism: Uses random projections or permutations to create signatures that preserve similarity.
• Common Variants: MinHash for Jaccard similarity; SimHash for cosine similarity.
• Use Case in Blocking: Efficiently groups high-dimensional embeddings (e.g., from text) into candidate pairs without exhaustive comparison.

The foundational probabilistic framework for record linkage, upon which many modern blocking and matching systems are built. It models the likelihood that two records refer to the same entity based on the pattern of agreements and disagreements across their attributes.

• Key Components: Defines match and non-match probabilities for each attribute.
• Linkage Rule: Uses the calculated likelihood ratio to classify pairs as matches, non-matches, or requiring clerical review.
• Relation to Blocking: The model's probability estimates can inform the selection of high-discriminatory attributes for creating effective blocking keys.

The process of converting data into a standard, consistent format before blocking and matching. It creates a single authoritative representation (a canonical form) for values like names, addresses, or product codes.

• Purpose: Reduces variation, making blocking keys more reliable and comparisons more accurate.
• Common Techniques: Standardizing date formats, normalizing text case, expanding abbreviations, and applying business rules.
• Example: Converting "International Business Machines," "IBM," and "I.B.M." all to the canonical form "IBM" for consistent blocking.

A rule-based entity resolution method that declares records a match if they exactly agree on a predefined set of attributes or derived match keys. It is intrinsically linked to blocking strategies.

• Blocking Key as Rule: A deterministic blocking key (e.g., first 3 letters of surname + ZIP code) creates a strict partition. Records are only compared if they fall into the same block.
• Pros and Cons: Highly transparent and fast, but lacks flexibility for handling typographical errors or missing data.
• Contrast with Probabilistic: Does not calculate likelihoods; uses Boolean logic for decision-making.

A graph-based operation critical for resolving entity clusters after pairwise matching within blocks. It infers that if record A matches B and B matches C, then A also matches C, ensuring logical consistency in the final entity set.

• Graph Representation: Records are nodes; match decisions are edges. Transitive closure finds all connected components in this graph.
• Challenge with Blocking: If blocking is too restrictive, transitive links between records in different blocks may be missed, leading to under-merging.
• Solution: Iterative blocking or recall-oriented blocking strategies help capture these transitive relationships.

The fundamental metrics for evaluating the effectiveness of a blocking strategy within an entity resolution pipeline.

• Recall (Completeness): The fraction of all true matching record pairs that are placed into the same block for comparison. Low recall means true matches are never compared.
• Precision (Efficiency): The fraction of record pairs within a block that are true matches. Low precision means the block contains many non-matches, wasting compute resources.
• The Trade-off: Blocking design is an optimization problem balancing high recall (few missed matches) against acceptable precision (manageable comparison cost).