Similarity Score

A similarity score is a numerical value, typically normalized between 0 and 1, that quantifies the degree of likeness between two data records, strings, or vector representations. Its primary purpose in entity resolution is to provide an objective, automatable metric for determining whether two records refer to the same real-world entity. A score of 1 indicates perfect identity, while 0 indicates complete dissimilarity. These scores feed into downstream matching rules or machine learning models to make link/no-link decisions.

Effective similarity scores are normalized to a consistent, interpretable range (e.g., 0 to 1). This standardization is critical for:

• Combining scores from different attributes (e.g., name, address, date of birth) into a composite match score.
• Setting universal thresholds for match classification across an entire pipeline.
• Human interpretability, allowing data stewards to understand why a pair was linked or not. Without normalization, comparing a Jaccard similarity on tokens (range 0-1) with a Levenshtein distance on strings (range 0-N) would be meaningless.

Scores are calculated using specific algorithms chosen based on data type and error patterns:

• String-based: For textual attributes. Examples include Levenshtein Distance (edit distance), Jaccard Similarity (set overlap), and Cosine Similarity on character n-grams.
• Phonetic: For matching names by sound, using algorithms like Soundex or Metaphone, which generate codes for approximate phonetic matching.
• Vector-based: For comparing dense representations (embeddings) of text or entities, typically using Cosine Similarity or Euclidean Distance in a high-dimensional space.
• Set-based: For comparing categorical or multi-valued fields, using Jaccard or Overlap Coefficient.

In the Fellegi-Sunter model, the canonical probabilistic matching framework, similarity scores are transformed into agreement weights. The model calculates:

• m-probability: The probability that an attribute agrees given the records are a match.
• u-probability: The probability that an attribute agrees given the records are a non-match. The log-likelihood ratio of these probabilities generates a weight for each attribute. The sum of these weights across all compared attributes produces a composite match score, which is then compared to upper and lower thresholds to classify the pair as a match, non-match, or potential match requiring clerical review.

A raw similarity score requires a decision threshold to trigger an action. Setting this threshold is a core engineering challenge balancing precision and recall.

• High Threshold (e.g., >0.95): Yields high precision (few false matches) but lower recall (misses true matches with data errors).
• Low Threshold (e.g., >0.7): Yields high recall (finds most true matches) but lower precision (includes many false matches). Sophisticated systems use dual thresholds, creating a clerical review zone for scores in a middle range. Thresholds are typically tuned via labeled data and evaluation of the confusion matrix.

In machine learning-based entity resolution, similarity scores are not the final output but are used as engineered features for a classifier. A feature vector for a record pair might include:

• Individual similarity scores for name, address, phone, date_of_birth.
• Derived features like the minimum, maximum, or average score across attributes.
• A flag indicating if a phonetic code matches. These features train a model (e.g., logistic regression, random forest, or a Siamese network) to predict a binary match label, often outperforming simple thresholding on a composite score.

Similarity scores are used to unify customer records from disparate systems (CRM, ERP, support tickets). Deterministic matching on exact fields like email is combined with probabilistic matching using scores on names, addresses, and phone numbers to create a golden record. This prevents duplicate marketing and ensures a single customer view.

• Key Metrics: Scores often combine Jaccard similarity on tokenized names and Levenshtein distance on addresses.
• Business Impact: Directly reduces operational costs from failed deliveries and redundant communications.

E-commerce and retail platforms use similarity scoring to identify duplicate or near-identical product listings from multiple vendors. This involves comparing high-dimensional feature vectors derived from product titles, descriptions, images, and specifications.

• Technical Approach: Cosine similarity on text embeddings from a language model is standard. For blocking, Locality-Sensitive Hashing (LSH) groups candidate products for pairwise scoring.
• Outcome: Creates a clean, canonical catalog, improving search relevance and price comparison accuracy.

A critical, high-stakes application where similarity scores link patient records across hospitals, labs, and clinics. Strict privacy regulations often preclude using national identifiers, making scoring on demographic data essential.

• Attributes Scored: Name (with phonetic encoding like Metaphone for sound-alikes), date of birth, and medical record number fragments.
• Challenge: Requires extremely high precision to avoid dangerous medical errors, often using supervised models with active learning for human-in-the-loop verification.

Resolves author name ambiguity in bibliographic databases (e.g., Digital Bibliography & Library Project, Semantic Scholar). The same author name (e.g., "J. Smith") may refer to hundreds of individuals. Similarity scores compare contextual features.

• Features Used: Co-author networks, institutional affiliations, research field keywords, and journal titles.
• Method: A Siamese network can learn a similarity function between paper-author pairs, clustering publications into distinct author entities.

Identifies synthetic identity fraud or organized fraud rings by scoring similarity between applicant profiles and known fraudulent entities. Instead of seeking exact matches, systems look for suspiciously high similarity in partial, manipulated attributes.

• Application: New account applications are scored against a database of known fraud indicators (addresses, phone numbers, devices).
• Technique: Anomaly detection models flag applications where aggregate similarity to bad actors exceeds a threshold, despite no single exact match.

Core to building Enterprise Knowledge Graphs. Similarity scores align ambiguous textual mentions from documents ("Apple") to their correct referent (the company vs. the fruit) in a knowledge base. This is the entity linking task.

• Scoring Method: Compares the context vector of the mention with the description vector of each candidate knowledge base entity using cosine similarity.
• Integration: This resolved entity becomes a node in the graph, enabling Graph-Based RAG for factually grounded AI responses.

A set-based similarity measure defined as the size of the intersection divided by the size of the union of two sets. Commonly used for comparing tokenized text.

• Formula: J(A,B) = |A ∩ B| / |A ∪ B|
• Range: 0 (no overlap) to 1 (identical sets)
• Use Case: Comparing bag-of-words representations or categorical attribute sets in entity resolution.

A string metric measuring the minimum number of single-character edits (insertions, deletions, substitutions) required to change one string into another. The basis for edit-distance calculations.

• Example: 'kitten' → 'sitting' has a Levenshtein distance of 3.
• Normalization: Often converted to a similarity score by 1 - (distance / max(length1, length2)).
• Application: Detecting typos and minor variations in names, addresses, and product codes.

Measures the cosine of the angle between two non-zero vectors in an inner product space, indicating their orientation similarity regardless of magnitude.

• Formula: cos(θ) = (A·B) / (||A|| ||B||)
• Range: -1 (opposite) to 1 (identical direction), typically 0-1 for positive vectors.
• Primary Use: Comparing dense vector embeddings (e.g., from word2vec, BERT) for semantic similarity in entity resolution.

The foundational probabilistic framework for record linkage. It calculates match and non-match probabilities for record pairs based on attribute agreements and disagreements.

• Core Components: m-probability (agreement given a match) and u-probability (agreement given a non-match).
• Output: A composite match weight, often converted to a final similarity score.
• Significance: Enables the combination of multiple, imperfect matching attributes into a single probabilistic score.

A broad category of techniques for finding strings or records that are approximately, but not exactly, identical. It is the practical application of similarity scores.

• Techniques Include: Edit-distance algorithms, phonetic encoding (Soundex, Metaphone), and token-based similarity.
• Key Challenge: Balancing sensitivity to true variations with robustness against noise and errors.
• Implementation: Often deployed in data cleaning, deduplication, and search systems.

The fundamental evaluation metrics for entity resolution systems, assessing the quality of matches determined by similarity score thresholds.

• Precision: The fraction of declared matches that are correct. Measures purity. TP / (TP + FP)
• Recall: The fraction of all true matches that are successfully retrieved. Measures completeness. TP / (TP + FN)
• Trade-off: Increasing the similarity score threshold typically increases precision but decreases recall, and vice-versa.