Vector Index

HNSW constructs a multi-layered graph where each layer is a subset of the previous one, enabling extremely fast approximate nearest neighbor search. It is the most widely used algorithm in production vector databases due to its excellent balance of speed and recall.

• Mechanism: Starts search at the top (sparsest) layer and navigates down through increasingly dense graphs to find neighbors.
• Trade-off: Offers high query speed and recall but requires more memory to store the graph structure.
• Use Case: The default index in many systems (e.g., Weaviate, Qdrant) for general-purpose semantic search where low latency is critical.

IVF partitions the vector space into clusters (Voronoi cells) using a clustering algorithm like k-means. Search is then restricted to the nearest clusters, dramatically reducing the number of distance computations.

• Mechanism: An inverted file maps each cluster centroid to a list of vectors within that cluster.
• Trade-off: Faster than brute-force search but recall depends on the number of clusters (nlist) searched (nprobe).
• Use Case: Often combined with Product Quantization (IVF-PQ) for billion-scale datasets where memory efficiency is paramount, such as in Facebook's FAISS library.

Product Quantization is a compression technique, not a standalone index. It dramatically reduces memory footprint by splitting vectors into sub-vectors and quantizing each sub-space independently.

• Mechanism: Creates a codebook of centroids for each sub-space. A vector is represented by a short code of centroid indices.
• Trade-off: Enables billion-scale vector search in RAM by approximating distances, with a minor cost to accuracy.
• Use Case: Almost always used in conjunction with IVF (as IVF-PQ) for in-memory search of massive datasets where storing full vectors is prohibitive.

Scalar Quantization reduces the precision of each vector component (e.g., from 32-bit floats to 8-bit integers), cutting memory usage by 75% with minimal accuracy loss.

• Mechanism: Maps the range of values for each dimension to a smaller integer range. Distance calculations use lookup tables for speed.
• Trade-off: Simpler than PQ, offers a good memory/accuracy balance, but provides less compression than PQ.
• Use Case: A standard optimization in many databases (e.g., Pinecone, Milvus) to increase the number of vectors that can be held in memory, improving cache efficiency and throughput.

DiskANN is designed for scenarios where the vector dataset is too large for main memory. It keeps a compressed graph index in RAM and fetches full-precision vectors from SSD during search.

• Mechanism: Builds a graph similar to HNSW but optimized for asynchronous I/O and SSD access patterns.
• Trade-off: Enables search over trillion-scale datasets on a single machine by trading RAM for disk, with query latency in milliseconds.
• Use Case: Critical for enterprise applications with vast, constantly updating knowledge bases where loading everything into RAM is cost-prohibitive.

A Brute-Force or Flat index performs an exhaustive search, computing the distance between the query vector and every vector in the dataset. It provides perfect, exact results.

• Mechanism: No pre-built data structure for pruning; calculates all distances using metrics like cosine similarity or L2 distance.
• Trade-off: Guarantees 100% recall but has a linear time complexity (O(N)), making it impractical for large, real-time systems.
• Use Case: Serves as a ground truth baseline for evaluating approximate index accuracy. Used for small datasets (< 10K vectors) where accuracy is non-negotiable and latency is acceptable.

This is the foundational use case. A vector index enables semantic search by finding text chunks with similar meaning to a query, not just matching keywords. This retrieved context is then fed to a Large Language Model (LLM) in a Retrieval-Augmented Generation (RAG) pipeline, grounding the model's responses in factual, proprietary data to reduce hallucinations.

• Core Mechanism: Query and documents are converted into embeddings. The index finds the nearest document vectors to the query vector.
• Enterprise Impact: Allows LLMs to answer questions based on internal documentation, support tickets, or research papers without retraining.

Vector indexes power recommendation engines by modeling users and items (products, articles, media) in a shared embedding space. Similarity in this space predicts affinity.

• User-Item Matching: A user's embedding (based on past behavior) is used as a query to find the nearest item vectors.
• Item-to-Item Recommendations: "Customers who viewed this also viewed..." is implemented by finding the nearest neighbors to a given product's vector.
• Session-Based Recs: Real-time recommendations are generated by creating a vector for the current user session and performing a fast ANN lookup.

Identifying duplicate or linked records across disparate databases is a classic data cleaning challenge. Vector indexes solve this by finding near-identical embeddings.

• Process: Records (customer profiles, product listings, company names) are embedded. The index finds all vectors within a very small distance threshold, flagging potential duplicates.
• Advantage over Rules: Captures semantic similarity (e.g., "IBM" and "International Business Machines") and handles typos or formatting differences more robustly than string-matching rules.
• Scale: Enables deduplication across millions or billions of records efficiently.

By learning a vector representation of "normal" behavior, vector indexes can help identify outliers that may indicate fraud, system intrusion, or operational failure.

• Modeling Normalcy: Embeddings are created for legitimate transactions, network events, or machine sensor readings. These form a dense cluster in vector space.
• Detection Query: A new event is embedded and queried against the index. If its nearest neighbors are far away (high distance), it is flagged as an anomaly.
• Dynamic Baselines: The index can be updated continuously to adapt to evolving patterns of normal behavior.

Vector indexes enable search across different data modalities by aligning them into a unified embedding space. A query in one modality can retrieve results in another.

• Image-to-Text / Text-to-Image: Search a photo database using a descriptive text query (e.g., "red sports car"), or find captions for a given image.
• Audio & Video Search: Find video clips or audio segments relevant to a text query by encoding all media into comparable vectors.
• Technical Foundation: Requires a multi-modal embedding model (e.g., CLIP) trained to place semantically similar text and images close together, which the index then queries.

Vector indexes enable low-latency pattern matching for event streams, triggering alerts when similar past incidents are detected.

• Streaming Context: Incoming log entries, security alerts, or customer support messages are converted to vectors in real-time.
• Proactive Alerting: The new vector is queried against an index of historical incident vectors. If a close match to a prior critical event is found, an alert is triggered before the situation escalates.
• Use Cases: IT operations (matching current error to known outages), cybersecurity (identifying attack patterns), and customer experience (detecting recurring complaint themes).