Vector Search

Unlike keyword search, vector search finds data based on conceptual meaning. It uses distance metrics like cosine similarity or Euclidean distance to measure proximity between vector embeddings. This allows it to retrieve relevant documents even when they don't share exact query terms.

• Example: A query for "canine companion" can retrieve documents about "dogs" and "pets".
• Core Mechanism: An embedding model (e.g., a transformer encoder) maps text, images, or other data into a dense vector space where semantic relationships are preserved geometrically.

Exact nearest neighbor search in high dimensions is computationally prohibitive. ANN algorithms trade a small amount of precision for massive gains in speed and scalability, making vector search practical for large datasets.

Key ANN algorithms include:

• HNSW (Hierarchical Navigable Small World): A graph-based method offering an excellent trade-off between recall, speed, and memory.
• IVF (Inverted File Index): Partitions the vector space into clusters and only searches the most promising ones.
• Product Quantization (PQ): Compresses vectors to reduce memory footprint, enabling billion-scale searches in RAM.

A vector index is a data structure optimized for storing embeddings and executing ANN queries. It is the core of any vector database or search library. Index choice directly impacts the system's performance profile.

• Flat Index: Stores all vectors for exact search; used for small datasets or as a ground truth benchmark.
• Composite Indexes: Combine methods like IVF-PQ to cluster, compress, and search efficiently.
• Dynamic Indexing: Supports real-time updates (inserts, deletes) without full rebuilds, crucial for production systems.

Vector search operates on dense embeddings, typically 384 to 1536 dimensions, where each dimension represents a latent feature learned by a model. These are distinct from sparse vectors used in TF-IDF or BM25.

• Generation: Created by embedding models like Sentence-BERT, OpenAI's text-embedding-ada-002, or E5 models.
• Properties: Similar concepts have small distances (high cosine similarity). The vector space captures analogies and hierarchies.
• Unified Representation: Enables multi-modal search (e.g., searching images with text) by mapping different data types into a shared embedding space.

Pure vector search can miss exact keyword matches or be misled by domain-specific jargon. Hybrid retrieval combines it with sparse, lexical search (e.g., BM25) to improve both recall and precision.

• Score Fusion: Techniques like Reciprocal Rank Fusion (RRF) combine ranked lists from dense and sparse retrievers without complex score normalization.
• Two-Stage Retrieval: A fast vector/BM25 first stage fetches candidates, followed by a cross-encoder reranker for precise final ordering.
• Result: Systems benefit from semantic understanding and exact term matching, crucial for enterprise RAG.

Deploying vector search at scale requires specialized infrastructure for indexing speed, query latency, and memory management.

• Libraries & Engines: Faiss (Facebook AI Research), Annoy (Spotify), and vector database extensions for Elasticsearch and OpenSearch.
• Dedicated Vector Databases: Systems like Pinecone, Weaviate, and Qdrant are built from the ground up for vector operations, handling sharding, replication, and hybrid metadata filtering.
• Performance: Latency for ANN search is typically < 100ms even over millions of vectors, enabling real-time applications like chat and recommendation systems.

Identifying near-duplicate records or linking records that refer to the same real-world entity across disparate, messy datasets. Vector embeddings capture the semantic essence of a record, making matches robust to typographical variations, different formatting, or paraphrased descriptions.

• Customer Data: Unifying customer profiles from sales, support, and web analytics where names and addresses are entered differently.
• Product Catalogs: Merging supplier feeds for the same item described with varying titles and specifications.
• Content Moderation: Flagging user-generated content that is semantically similar to known prohibited material.

Modeling normal behavior as a region in vector space and flagging outliers. Transactions, user sessions, or system logs are embedded; items that are semantic outliers from the dense cluster of normal activity indicate potential fraud or failure.

• Financial Security: Detecting transaction patterns that are semantically distant from a user's typical behavior.
• IT Security: Identifying network traffic or log entries that deviate from the learned pattern of normal operations.
• Manufacturing: Spotting sensor readings from production equipment that are anomalous compared to vectors representing optimal operation.

Providing autonomous agents with a scalable, associative memory. The agent's experiences, observations, or learned facts are encoded into vectors and stored. The agent can later retrieve relevant past memories by performing a similarity search on its current state or goal vector.

• Conversational Agents: Recalling relevant details from past interactions with a user to maintain context.
• Game NPCs: Remembering past encounters with players or the state of the game world.
• Robotic Systems: Retrieving similar past situations and successful actions when faced with a new task or environment.

An embedding model is a neural network, typically a transformer encoder, trained to map discrete data like text into a continuous, high-dimensional vector space. Its core function is to ensure that semantic similarity between inputs corresponds to geometric proximity (e.g., short cosine distance) between their vector outputs. Examples include models like Sentence-BERT (SBERT) and text-embedding-ada-002. The quality of the embedding model directly determines the effectiveness of the entire vector search pipeline.

Approximate Nearest Neighbor search is a class of algorithms for efficiently finding the closest vectors to a query in high-dimensional space. Because exact search is computationally prohibitive for large datasets, ANN algorithms trade a small amount of precision for massive speed gains. Key algorithms include:

• HNSW (Hierarchical Navigable Small World): A graph-based method offering an excellent balance of recall, speed, and memory.
• IVF (Inverted File Index): Partitions the space into clusters and only searches the most promising ones.
• Product Quantization (PQ): Compresses vectors to drastically reduce memory footprint for billion-scale searches.

A dual encoder (or bi-encoder) is a neural network architecture used for efficient retrieval. It encodes two inputs—such as a query and a document—separately into fixed-dimensional vector embeddings. The relevance score is then computed via a simple, fast similarity measure like cosine similarity or dot product between the two vectors. This design enables pre-computation of all document embeddings, making it ideal for the first-stage retriever in a two-stage system. Dense Passage Retrieval (DPR) is a prominent example of this architecture.

Cosine similarity is the predominant metric for measuring the similarity between two vector embeddings in high-dimensional space. It calculates the cosine of the angle between the vectors, yielding a value between -1 and 1. A value of 1 indicates identical orientation (maximal similarity), 0 indicates orthogonality, and -1 indicates opposite direction. It is favored over Euclidean distance for text embeddings because it measures directional alignment, which often correlates better with semantic meaning, and is invariant to the magnitude (length) of the vectors.