Cosine Similarity

Vector search is the overarching retrieval paradigm that uses cosine similarity as one of its core comparison metrics. It finds items in a dataset by comparing their high-dimensional vector representations (embeddings).

• Core Operation: Transforms queries and documents into vectors and performs a nearest neighbor search.
• Primary Use Case: Enables semantic search, where results match the conceptual meaning of a query, not just keyword overlap.
• Infrastructure: Typically powered by specialized vector databases like Pinecone, Weaviate, or Qdrant, which are optimized for this operation.

k-Nearest Neighbors (k-NN) is the fundamental, exact search algorithm that vector search implementations approximate. For a given query vector, it finds the 'k' vectors in the dataset with the smallest distance (or highest similarity, like cosine).

• Brute-Force Nature: Computes the distance between the query and every vector in the dataset, guaranteeing perfect accuracy.
• Computational Cost: Complexity is O(N*d), where N is dataset size and d is dimensionality, making it impractical for large-scale production use.
• Baseline Utility: Serves as the ground-truth benchmark for evaluating faster, approximate methods.

Approximate Nearest Neighbor (ANN) search is a family of algorithms that trade a small, configurable amount of accuracy for orders-of-magnitude faster retrieval speeds on large vector datasets.

• Speed-Accuracy Trade-off: Uses intelligent indexing structures to avoid comparing the query to every vector. Common algorithms include HNSW, IVF, and LSH.
• Production Necessity: Essential for real-time retrieval from indexes containing millions or billions of vectors.
• Key Metric: Measured by recall@k, which evaluates what percentage of the true k-nearest neighbors are found by the approximate search.

Euclidean distance (L2 norm) is the other primary metric for comparing vectors, measuring the straight-line distance between two points in space. It serves a different purpose than cosine similarity.

• Mathematical Definition: sqrt(Σ (A_i - B_i)²). It is sensitive to both the direction and the magnitude of vectors.
• When to Use: Ideal for use cases where vector magnitude carries meaningful information, such as comparing embeddings from models where the norm correlates with confidence or signal strength.
• Contrast with Cosine: For normalized vectors (unit length), Euclidean distance and cosine similarity are monotonically related: a smaller Euclidean distance corresponds to a larger cosine similarity.

The dot product (or inner product) is the un-normalized mathematical operation at the heart of cosine similarity. For vectors A and B, it is calculated as Σ (A_i * B_i).

• Relationship to Cosine: Cosine similarity is the dot product of L2-normalized vectors: cos(A, B) = (A · B) / (||A|| * ||B||).
• Maximum Inner Product Search (MIPS): A critical retrieval problem focused on finding vectors with the highest dot product to a query. This is not equivalent to nearest neighbor search under cosine or Euclidean unless vectors are normalized.
• Key Application: Fundamental to attention mechanisms in transformers and recommendation systems where un-normalized scores represent affinity.

Hybrid search is a retrieval strategy that combines the strengths of vector search (semantic, using metrics like cosine similarity) with sparse retrieval (keyword-based, using models like BM25).

• Motivation: Mitigates the weaknesses of each approach. Vector search can miss exact keyword matches, while keyword search fails on semantic paraphrasing.
• Fusion Method: Results from both retrieval paths are combined using algorithms like Reciprocal Rank Fusion (RRF) or weighted score summation.
• Outcome: Produces a final ranked list with higher recall and precision, ensuring both semantically relevant and keyword-precise documents are retrieved.