Cosine Similarity

Cosine similarity is magnitude-agnostic, meaning it focuses solely on the direction of vectors, not their length. This is critical in NLP because document length or word frequency should not dominate the similarity score. Two documents with identical word distribution but different lengths will have a cosine similarity of 1.0, as their vectors point in the same direction. This property makes it superior to Euclidean distance for text, where raw term frequency can vary wildly.

The output is strictly bounded between -1 and 1, providing an intuitive, normalized scale.

• 1: Vectors are perfectly aligned (identical direction).
• 0: Vectors are orthogonal (no correlation).
• -1: Vectors are diametrically opposed. In dense retrieval with modern embeddings, values typically range from 0 to 1 because embeddings are often non-negative, making the angle always less than or equal to 90 degrees.

For unit-normalized vectors (where the magnitude is 1), cosine similarity reduces to a simple dot product. This is a massive optimization for vector search. Instead of calculating the full cosine formula, the system computes A · B. This allows Maximum Inner Product Search (MIPS) algorithms to run extremely fast on GPUs, enabling billion-scale similarity searches in milliseconds.

Cosine similarity measures the cosine of the angle θ between two vectors. If the angle is small, the cosine is close to 1, indicating high similarity. This geometric view helps in understanding why it works for embeddings: the model is trained to place semantically similar items in the same narrow cone of the embedding space. The metric effectively answers: 'How sharp is the angle between these two concepts in high-dimensional space?'

While Euclidean distance measures the straight-line distance between two points, cosine similarity measures orientation. Consider two documents where one is a verbatim copy of the other but repeated twice. Their Euclidean distance would be large due to the magnitude difference, but their cosine similarity would be 1.0. For semantic search, orientation (topic) is usually more important than magnitude (length), making cosine similarity the default choice.

Cosine similarity is the scoring function inside contrastive loss and InfoNCE loss. During training, the model computes the cosine similarity between a query and a positive passage (targeting 1.0) and between the query and negative passages (targeting 0.0). The loss function then penalizes the model proportionally to how poorly it separates these scores, directly shaping the embedding space to make cosine similarity a reliable semantic metric.

The dot product measures both magnitude and angle, while cosine similarity normalizes vectors to unit length, isolating pure directional alignment. In dense retrieval, cosine similarity is preferred when embedding magnitudes vary due to passage length. The relationship is: cosine(A,B) = dot(A,B) / (||A|| * ||B||). Use dot product when magnitude encodes meaningful signal like term frequency; use cosine when only semantic orientation matters.

Euclidean distance measures straight-line proximity in vector space, while angular distance (derived from cosine similarity) measures orientation difference. For L2-normalized embeddings, these metrics become monotonic to each other. Key distinction: Euclidean distance is sensitive to vector magnitudes; angular distance is not. In high-dimensional spaces, angular distance often better captures semantic relationships because the curse of dimensionality degrades Euclidean distance's meaningfulness.

L2 normalization projects vectors onto the unit hypersphere, making cosine similarity equivalent to dot product. This is standard practice in models like Sentence-BERT and DPR. Benefits include: faster computation since dot product replaces cosine, improved training stability, and meaningful similarity scores bounded in [-1, 1]. Without normalization, cosine similarity must compute vector magnitudes at query time, adding latency to retrieval pipelines.

Setting a minimum cosine similarity threshold filters irrelevant results before re-ranking. Typical thresholds range from 0.7 to 0.85 depending on domain specificity. Too low: noise overwhelms the re-ranker. Too high: recall collapses. Dynamic thresholding adapts per query by analyzing score distribution statistics. Some systems use relative thresholding, keeping the top-N results above a baseline similarity floor.

Approximate Nearest Neighbor libraries like FAISS and ScaNN natively support cosine similarity via inner product search on normalized vectors. HNSW graphs use cosine distance as an edge-weight metric during construction. When building production indexes, pre-normalize embeddings to convert cosine similarity into inner product, which is hardware-accelerated via BLAS libraries and GPU kernels. This optimization reduces query latency by 2-5x.

Contrastive learning objectives like NT-Xent loss (used in SimCLR and SimCSE) operate on cosine similarity between normalized embeddings. The loss maximizes cosine similarity for positive pairs while minimizing it for negative pairs within a temperature-scaled softmax. The temperature parameter controls the concentration of the distribution: lower temperatures sharpen distinctions, making the model more sensitive to hard negatives.