Duplicate Detection

Applies exact-match rules to predefined identifier fields such as Medical Record Number (MRN), Social Security Number, or a composite key of first name, last name, and date of birth.

• Produces binary outcomes: match or no match
• Zero false positives when using a verified unique identifier
• Fails silently when data contains typographical errors, transpositions, or formatting inconsistencies
• Example: Two records with MRN A12345 and A12345 are deterministically linked

Uses approximate string comparison algorithms to compute similarity scores between field values, generating a probability that two records refer to the same entity.

• Levenshtein distance measures edit operations required to transform one string into another
• Jaro-Winkler gives higher weight to matching prefixes, ideal for names
• Phonetic algorithms like Soundex and Metaphone normalize names by pronunciation
• Example: Jon Smith and John Smyth score 0.92 on Jaro-Winkler, exceeding a 0.85 threshold

Reduces the O(n²) computational complexity of comparing every record against every other record by partitioning the dataset into candidate blocks before detailed comparison.

• Sorted Neighborhood Method slides a window over sorted records, comparing only nearby entries
• Canopy clustering uses cheap, approximate metrics to create overlapping blocks
• TF-IDF indexing treats records as documents to retrieve candidate matches via inverted indices
• Reduces pairwise comparisons from billions to millions in large MPI datasets

Assigns discriminative power weights to different attributes based on their uniqueness and reliability, rather than treating all field matches equally.

• Date of Birth typically carries high weight due to low collision probability
• Last Name weighted higher than First Name due to lower variability
• ZIP Code receives moderate weight but decays over time as patients relocate
• A composite score is calculated: Σ(weight_i × field_similarity_i)
• Example: A DOB match contributes 0.30 to the total score, while a gender match contributes only 0.05

Resolves multi-record linkage chains where record A matches B, and B matches C, but A and C fall below the direct match threshold. Transitive closure logically deduces that all three represent the same entity.

• Prevents fragmented golden records where a single patient has multiple partial profiles
• Requires careful conflict resolution when transitive links contain contradictory attribute values
• Often implemented using graph-based clustering algorithms like connected components
• Critical for longitudinal patient record linkage across multiple facility visits

Defines the deterministic logic for merging matched duplicate records into a single golden record by selecting the most trustworthy value for each attribute from among the source records.

• Recency rules select the most recently updated value
• Source authority rules prefer data from EHR systems over patient-entered portals
• Completeness rules favor non-null values over missing data
• Example: When merging two records with conflicting phone numbers, the system selects the value from the record with the most recent last_updated timestamp

A string comparison technique that identifies non-exact matches between text fields by calculating similarity scores rather than requiring binary equality. Essential for duplicate detection when records contain typographical errors, abbreviations, or formatting inconsistencies.

Common algorithms include:

• Levenshtein distance: Edit distance measuring insertions, deletions, and substitutions
• Jaro-Winkler: Optimized for name matching with a prefix bonus
• Phonetic algorithms: Soundex and Metaphone for homophone detection
• n-gram similarity: Breaking strings into character subsequences for comparison

Fuzzy matching thresholds must be carefully tuned to balance precision and recall.

The single, best-surviving version of a data entity created by merging and cleansing duplicate records to provide a unified, authoritative 360-degree view of a subject. Golden record creation is the ultimate objective of duplicate detection workflows.

Key characteristics:

• Survivorship rules: Logic determining which value prevails when duplicates conflict
• Lineage tracking: Preserving provenance of each merged field
• Continuous curation: Golden records require ongoing stewardship as new duplicates emerge

In healthcare, a golden patient record prevents fragmented clinical histories and reduces medical errors from incomplete information.

A rule-based approach to duplicate detection that applies exact or predefined matching criteria to determine record equivalence. Unlike probabilistic methods, deterministic matching produces the same result for identical inputs every time.

Common deterministic strategies:

• Exact match on unique identifiers: Social Security Number, MRN, or National Provider Identifier
• Composite key matching: Combining multiple fields like DOB + ZIP + Last Name
• Rule cascades: Applying increasingly lenient match rules in sequence

Deterministic matching is computationally efficient and auditable but may miss duplicates caused by data entry errors or missing identifiers.

A statistical framework for duplicate detection that assigns match weights to field comparisons based on their discriminatory power. Developed by Fellegi and Sunter in 1969, this approach calculates the likelihood that two records represent the same entity.

Core components:

• m-probability: Chance that a field matches given the records are a true match
• u-probability: Chance that a field matches by random coincidence
• Match weight: Log-likelihood ratio combining agreement and disagreement evidence
• Threshold classification: Scores above upper threshold are matches; below lower threshold are non-matches; between requires clerical review

This method handles missing data and partial agreement gracefully.

Performance optimization techniques that reduce the quadratic complexity of pairwise record comparison by partitioning the dataset into candidate subsets. Without blocking, comparing every record against every other becomes computationally prohibitive.

Common blocking strategies:

• Sorted neighborhood: Sorting records by a blocking key and sliding a fixed-size window
• Canopy clustering: Using cheap, approximate distance measures to create overlapping clusters
• Q-gram indexing: Building inverted indices on character n-grams for fast retrieval
• Phonetic blocking: Grouping records by Soundex or Metaphone codes for name matching

Effective blocking dramatically reduces the number of comparisons while maintaining high recall of true duplicates.