Entity Reconciliation

Employs string similarity metrics like Levenshtein distance, Jaro-Winkler, and phonetic algorithms (Soundex, Metaphone) to tolerate typographical errors, abbreviations, and formatting inconsistencies. Unlike exact matching, fuzzy logic calculates a similarity score between 0 and 1.0, allowing a threshold-based decision for a match.

• Levenshtein Distance: Counts single-character edits (insertions, deletions, substitutions).
• Jaro-Winkler: Favors strings that match from the beginning, ideal for proper names.
• Phonetic Hashing: Converts strings to a code representing pronunciation, catching 'Stefan' vs. 'Stephen'.

A computational efficiency technique that avoids the impossible task of comparing every local record against every knowledge base entry (an O(n*m) problem). Blocking keys partition datasets into mutually exclusive blocks using cheap heuristics like the first three letters of a name or a Soundex code. Only records within the same block are compared.

• Sorted Neighborhood: Sorts records by a key and slides a fixed-size window over the list.
• Canopy Clustering: Uses a cheap, high-recall metric to create overlapping clusters before applying an expensive, high-precision metric.

For each local record, the reconciliation service queries the external knowledge base API to retrieve a shortlist of candidate entities. Each candidate is assigned a reconciliation score reflecting the confidence of the match. This score is often a composite of multiple feature similarities, including name matching, attribute agreement (e.g., birth date, location), and structural context.

• Vector Space Models: Modern systems use dense entity embeddings to find semantically similar candidates via approximate nearest neighbor (ANN) search.
• Feature Vectors: Traditional systems construct a binary vector of matching attributes and train a logistic regression classifier.

Entity reconciliation is semi-automated by design. Matches above a high-confidence threshold are processed automatically, while those in a grey zone are routed to a human curator for judgment. This interface allows users to confirm, reject, or manually search for the correct entity.

• Active Learning: The system learns from human corrections to improve future matching accuracy.
• Judgment History: All human decisions are logged to create an auditable trail and a training dataset for model refinement.

Before matching, local columns must be mapped to the target knowledge base's schema properties. A 'Company Name' column is mapped to the schema:legalName property. Type constraints further refine the search by restricting candidates to a specific class, such as Q4830453 (business) or Q5 (human), preventing a person's name from matching a geographical location.

• Property Mapping: Aligns local attributes with knowledge graph predicates.
• Type Filtering: Narrows the candidate pool to entities of a specific ontological type.

Reconciliation is typically executed via a RESTful API that accepts batches of records in JSON or CSV format. Services like the Wikidata Reconciliation API and tools like OpenRefine standardize this workflow. The API handles rate limiting, asynchronous processing for large datasets, and returns structured match results with persistent entity IDs.

• Endpoint: Standardized endpoints like /reconcile accept queries and return candidates.
• Asynchronous Mode: Large batches are processed in the background with a callback or polling mechanism.

The statistical process of identifying and joining records across different datasets that correspond to the same real-world entity when a reliable unique identifier is absent. Unlike reconciliation against an external knowledge base, record linkage typically operates between internal databases—for example, merging customer records from a CRM and an ERP system. The Fellegi-Sunter model provides the probabilistic framework, calculating match weights based on attribute agreement and disagreement patterns across fields like name, address, and date of birth.

A computational efficiency technique that partitions datasets into mutually exclusive blocks using cheap heuristics before pairwise comparison. Without blocking, reconciling two datasets of 100,000 records each would require 10 billion comparisons. Blocking reduces this dramatically by only comparing records that share a common property:

• Phonetic blocking: Group by Soundex or Metaphone encoding of surnames
• N-gram blocking: Cluster by shared character trigrams
• Sorted neighborhood: Sort by a key and slide a fixed-size window Effective blocking trades a small amount of recall for massive speed gains.

String comparison techniques that calculate similarity scores between text strings to identify non-exact duplicates. Essential for reconciliation because real-world data contains typographical errors, abbreviations, and formatting inconsistencies. Common algorithms include:

• Levenshtein distance: Minimum edit operations to transform one string into another
• Jaro-Winkler: Optimized for short strings like names, weighting prefix matches
• Token-based: Jaccard or cosine similarity over word n-grams
• Phonetic: Soundex or Double Metaphone for names that sound alike Most reconciliation services combine multiple fuzzy metrics with weighted thresholds.

The initial retrieval phase in entity linking and reconciliation that produces a shortlist of possible matches from the target knowledge base for each query entity. This step uses a surface form dictionary or approximate nearest neighbor search over entity embeddings to avoid scanning millions of entries. For example, reconciling 'J. S. Bach' might generate candidates including Johann Sebastian Bach (Q1339), Johann Christian Bach (Q106641), and Johann Christoph Bach (Q372024). The quality of candidate generation directly bounds the ceiling of final reconciliation accuracy.

The broader data management discipline of accurately matching and merging identity data across disparate systems to create a unified, 360-degree view of a customer, patient, or organization. Identity resolution extends beyond simple record linkage by incorporating:

• Temporal awareness: Handling name changes, address moves, and life events
• Relationship graphs: Leveraging known associations to resolve ambiguous matches
• Survivorship rules: Determining which conflicting attribute values to retain in the golden record Enterprise Master Data Management (MDM) platforms operationalize identity resolution at scale.

The specific application of record linkage techniques to find and merge duplicate records within a single dataset to create a clean master record. While reconciliation matches against an external reference, deduplication is an internal cleansing operation. Common scenarios include:

• Removing duplicate customer profiles from a CRM after a merger
• Consolidating product SKUs entered with slight variations
• Cleaning author lists in bibliographic databases Deduplication typically uses transitive closure: if A matches B and B matches C, then A, B, and C form a single cluster despite A and C not directly matching.