Cosine Similarity

Cosine similarity measures the cosine of the angle between two vectors in a multi-dimensional space. It is defined as the dot product of the vectors divided by the product of their magnitudes (L2 norms).

• Formula: cos(θ) = (A · B) / (||A|| * ||B||)
• Range: Outputs a value between -1 and 1.
• Interpretation: A value of 1 indicates identical direction (angle of 0°), 0 indicates orthogonality (90°), and -1 indicates opposite direction (180°).

This focus on orientation, not magnitude, is its defining characteristic.

A core feature is its insensitivity to vector magnitude. It assesses similarity based purely on the direction of the vectors, not their length.

• Example: In text analysis, the documents "The cat sat." and "The cat sat on the mat." will have similar embeddings. Even though the second vector has a larger magnitude (more words), their cosine similarity can be high because the direction of their semantic meaning is aligned.
• Benefit: This makes it robust for comparing documents of different lengths or data points with varied scales, a common scenario in real-world datasets.

Cosine similarity is computationally efficient and well-behaved in high-dimensional spaces, such as those created by modern embedding models (e.g., 384, 768, or 1536 dimensions).

• Curse of Dimensionality: In high dimensions, Euclidean distance becomes less meaningful as most points become equidistant. Cosine similarity, focusing on angle, remains a stable metric.
• Use Case: It is the standard similarity measure for nearest neighbor search in vector databases when using dense embeddings from models like sentence-transformers or OpenAI's embeddings API.

It is the de facto metric for comparing text embeddings generated by transformer models, forming the backbone of semantic search and retrieval-augmented generation (RAG).

• Semantic Search: Queries and documents are converted to vectors. The system retrieves documents with the highest cosine similarity to the query vector.
• Clustering & Recommendations: Used in algorithms like spherical k-means (which clusters based on vector direction) and for building content-based recommendation systems.
• Practical Note: For purely positive-valued vectors (like TF-IDF or many deep learning embeddings), the similarity range is typically between 0 and 1.

Cosine similarity is mathematically related to, but distinct from, other common distance and similarity measures.

• Euclidean Distance: Measures the straight-line distance between vector points. For L2-normalized vectors (unit vectors), Euclidean distance and cosine similarity are inversely related: Euclidean Distance² = 2 * (1 - Cosine Similarity).
• Dot Product: The numerator of the cosine formula. Without normalization by magnitude, the dot product is sensitive to both direction and length.
• Choosing a Metric: Use cosine similarity when direction matters most; use Euclidean distance when absolute magnitude differences are critical.

Effective use requires understanding its implementation, often involving a pre-processing step of L2 normalization.

• L2 Normalization: Scaling each vector to have a unit norm (magnitude of 1). After this step, cosine_similarity(A, B) = dot_product(A_normalized, B_normalized). This simplifies computation and storage.
• Computational Form: For large-scale comparisons, such as in vector databases, the similarity is often calculated using optimized linear algebra libraries that compute normalized dot products.
• Thresholding: In entity resolution, a similarity threshold (e.g., 0.8 or 0.9) is applied to decide if two vectorized records represent a match.

A similarity score is a numerical value, typically normalized between 0 and 1, that quantifies the degree of likeness between two data points, records, or vectors. In entity resolution, it is the fundamental output of comparison functions.

• Purpose: Drives match/no-match decisions, often used as input to a downstream classification model or threshold rule.
• Calculation: Can be derived from various metrics like cosine similarity, Jaccard index, or Levenshtein distance.
• Aggregation: For multi-attribute records, individual attribute similarity scores are often combined (e.g., weighted average) to produce an overall record similarity score.

Jaccard similarity (or Jaccard index) measures the similarity between two finite sets. It is defined as the size of the intersection divided by the size of the union of the sets.

• Formula: J(A, B) = |A ∩ B| / |A ∪ B|
• Use Case: Ideal for comparing tokenized text (e.g., product descriptions, bag-of-words models) or categorical data represented as sets.
• Contrast with Cosine: While cosine similarity operates on vectors with magnitudes, Jaccard is purely set-based and ignores element frequency or weight. For binary vector representations, Jaccard and cosine similarity are related but not identical.

A vector embedding is a dense, low-dimensional numerical representation of data (like text, an image, or an entity) in a continuous vector space. Cosine similarity is most meaningfully applied to these representations.

• Creation: Generated by models like Word2Vec, Sentence Transformers, or foundation model APIs.
• Property: Semantically similar items have embeddings that are close in the vector space, as measured by high cosine similarity.
• Role in ER: In modern entity resolution, records are often converted to embeddings (e.g., of a concatenated description field). Matching then becomes a nearest neighbor search in this embedding space using cosine similarity as the distance metric.

Locality-Sensitive Hashing is a technique that hashes input items so that similar items map to the same hash 'buckets' with high probability. It is a critical performance optimization for similarity search.

• Purpose: Approximates nearest neighbor search in sublinear time, avoiding the quadratic cost of comparing all pairs. This is essential for blocking in large-scale entity resolution.
• Mechanism: Uses families of hash functions designed to preserve similarity (e.g., SimHash for cosine similarity).
• Trade-off: Introduces a tunable balance between recall (finding true matches) and computational efficiency.

Probabilistic matching is a statistical entity resolution method that calculates the likelihood that two records refer to the same entity. Cosine similarity often serves as a key feature within this framework.

• Foundation: Based on the Fellegi-Sunter model, which estimates probabilities for attribute agreements given a match or non-match.
• Process: Combines evidence from multiple, possibly imperfect, fields (e.g., name, address, date of birth) to compute a composite match probability.
• Integration: A cosine similarity score on a text field (like 'company name') can be converted into a likelihood ratio and incorporated into the overall probabilistic score.

A Siamese network is a neural architecture that uses two or more identical subnetworks to process different input samples. Its goal is to learn a representation where similarity can be computed directly (e.g., via cosine distance).

• Structure: The twin networks share the same weights and parameters. They output embeddings for their respective inputs.
• Training: Trained on pairs of records (match/non-match) using a contrastive or triplet loss function, which teaches the network to pull similar items closer and push dissimilar items apart in the embedding space.
• Output: The cosine similarity between the two output embeddings becomes a highly tuned, task-specific similarity score for entity resolution.