Semantic Similarity

The core principle that words appearing in similar contexts tend to have similar meanings. Modern systems operationalize this by training neural networks to predict context words, producing dense vector representations where semantic proximity is encoded as geometric distance.

• Static embeddings (Word2Vec, GloVe) assign a single vector per word
• Contextualized embeddings (BERT, ELMo) generate dynamic vectors based on surrounding words
• The cosine of the angle between two vectors serves as the similarity score

Semantic similarity is computed as a mathematical function over high-dimensional embedding spaces, typically using cosine similarity or Euclidean distance. These operations enable analogical reasoning and efficient nearest-neighbor search.

• Cosine similarity: Measures the cosine of the angle between two vectors, ranging from -1 (opposite) to 1 (identical)
• Dot product: Used in attention mechanisms to compute alignment scores between query and key vectors
• Manhattan distance: Occasionally used when sparsity is a desired property in the representation

An alternative to distributional methods, this approach measures similarity by traversing structured ontologies and semantic networks. The shorter the path between two entity nodes, the higher their conceptual relatedness.

• Wu & Palmer similarity uses the depth of the lowest common subsumer in a taxonomy
• Leacock-Chodorow normalizes path length by the overall taxonomy depth
• Resnik similarity incorporates information content based on corpus frequency of the shared ancestor

Two architectural paradigms govern how similarity is computed. Cross-encoders process both text spans jointly through full self-attention, yielding high accuracy at the cost of speed. Bi-encoders encode each span independently, enabling pre-computation and fast cosine comparison.

• Cross-encoders are used for re-ranking top candidates retrieved by a bi-encoder
• Bi-encoders power dense retrieval over millions of documents via approximate nearest neighbor search
• The trade-off is between computational cost and pairwise interaction depth

Specialized architectures trained with contrastive or triplet loss to produce sentence-level embeddings where semantically similar pairs are pulled together and dissimilar pairs are pushed apart in vector space.

• Siamese BERT-Networks (SBERT) fine-tune BERT using siamese and triplet network structures
• Training data consists of positive pairs (paraphrases, entailments) and negative pairs (contradictions, random samples)
• The resulting embeddings can be compared directly using cosine similarity without cross-attention

Evaluation is standardized through the STS Benchmark and related datasets, which provide human-annotated similarity scores on a continuous scale from 0 (completely unrelated) to 5 (semantically equivalent).

• STS-B contains sentence pairs from news headlines, captions, and forums
• SICK (Sentences Involving Compositional Knowledge) tests logical inference alongside similarity
• Pearson and Spearman correlation coefficients measure alignment between model predictions and human judgments

The most common metric for semantic similarity in dense vector spaces. It measures the cosine of the angle between two embedding vectors, providing a magnitude-invariant score ranging from -1 (opposite) to 1 (identical).

• Formula: cos(θ) = (A·B) / (||A|| ||B||)
• Use Case: Comparing document embeddings in a vector database.
• Key Insight: Focuses purely on orientation, ignoring the absolute length of the document vector.

Measures the straight-line distance between two points in a high-dimensional embedding space. Unlike cosine similarity, it is sensitive to the magnitude of the vectors.

• Behavior: A smaller distance indicates higher similarity.
• Application: Often used in clustering algorithms like K-Means.
• Contrast: Use Euclidean distance when vector magnitude (e.g., word frequency emphasis) matters; use cosine similarity when only direction matters.

A fundamental algebraic operation that multiplies corresponding elements of two vectors and sums the results. In normalized embeddings, the dot product is equivalent to cosine similarity.

• Efficiency: Computationally faster than cosine similarity as it avoids calculating vector magnitudes.
• Usage: The default scoring mechanism in many approximate nearest neighbor (ANN) search libraries for maximum throughput.

A high-precision alternative to cosine similarity for semantic textual similarity (STS). Instead of encoding sentences independently, a Cross-Encoder processes the query-document pair simultaneously through full self-attention.

• Architecture: Often a BERT-style model with a classification head.
• Trade-off: Extremely accurate but computationally prohibitive for initial retrieval; typically used as a re-ranker over top-K results from a Bi-Encoder.

A symbolic approach to semantic similarity that measures the conceptual distance between two entities in a structured ontology or knowledge graph.

• Wu & Palmer: Scores similarity based on the depth of the lowest common subsumer (LCS) in the taxonomy hierarchy.
• Leacock & Chodorow: Uses the shortest path length normalized by the maximum depth of the taxonomy.
• Advantage: Provides deterministic, explainable similarity scores based on curated human knowledge.

Calculates distance as the sum of the absolute differences of the Cartesian coordinates. In high-dimensional sparse vectors, Manhattan distance can be more meaningful than Euclidean distance.

• Formula: Σ |Ai - Bi|
• Robustness: Less sensitive to outliers in specific dimensions compared to the squared penalty of Euclidean distance.
• Context: Often used in sparse retrieval contexts or when feature dimensions are independent.