FAISS (Facebook AI Similarity Search)

FAISS uses a building-block architecture where indexes are constructed by chaining pre-processing steps, coarse quantizers, and fine quantizers. This allows engineers to tailor the index to specific accuracy, speed, and memory constraints.

• Pre-processing: Includes PCA (Principal Component Analysis) for dimensionality reduction and L2 normalization.
• Coarse Quantizer: The first-level index (e.g., IVF) that selects a subset of candidate vectors.
• Fine Quantizer: The second-level refinement (e.g., PQ) that computes exact or more accurate distances within the candidate list.
• Example: An IndexIVFPQ combines an IVF coarse quantizer with a Product Quantization fine quantizer.

FAISS is optimized for both high-throughput batch processing and low-latency single queries, making it suitable for offline indexing jobs and real-time inference serving.

• Batched Queries: Uses matrix multiplication and optimized memory layouts to process thousands of query vectors simultaneously, maximizing GPU/CPU utilization.
• Single Query: Employs efficient search paths and cache-aware algorithms to minimize latency for online applications.
• Distance Computations: Supports multiple metrics including L2 (Euclidean) distance, inner product, and cosine similarity (via L2 normalization).

FAISS provides direct control over memory-mapped indices and serialization, enabling the use of indices larger than available RAM and efficient persistence to disk.

• Memory Mapping: An index can be read directly from disk without loading entirely into RAM, with the operating system handling page caching.
• Serialization: Supports saving complete index state (including trained centroids and encoded vectors) to a file and reloading it.
• Clone & Merge: Functions to copy indices between CPU/GPU and merge separate indices into a single one, facilitating distributed index construction.

While FAISS is a library, not a standalone database server, it is designed to integrate seamlessly into larger multimodal data architectures.

• Input/Output: Operates on in-memory numpy arrays, making it compatible with Python data science stacks (NumPy, PyTorch, TensorFlow).
• Metadata Handling: FAISS returns vector IDs; it is typically paired with a key-value store or database (like PostgreSQL) to retrieve the original metadata (text, image paths, etc.) associated with those IDs.
• Hybrid Search: Can be combined with keyword filters (post-search filtering) implemented in the application layer to create hybrid retrieval systems.

An Approximate Nearest Neighbor (ANN) index is a data structure that enables fast, but not perfectly accurate, similarity search in high-dimensional spaces by trading off some precision for significant gains in query speed and memory efficiency. FAISS provides implementations of several ANN algorithms. Key trade-offs include:

• Search Speed vs. Recall: Faster indexes often return slightly less accurate results.
• Memory Usage vs. Accuracy: More memory-intensive indexes can achieve higher precision.
• Index Build Time: Some algorithms are faster to construct than others. Common ANN methods include Inverted File (IVF), Product Quantization (PQ), and Hierarchical Navigable Small World (HNSW) graphs.

Hierarchical Navigable Small World (HNSW) is a graph-based algorithm for constructing an Approximate Nearest Neighbor (ANN) index, known for its high search speed and recall accuracy. It organizes vectors into a multi-layered graph structure where:

• The bottom layer contains all data points.
• Higher layers are subsets of the layer below, creating a navigable "highway" system.
• Search begins at the top layer and greedily traverses down, quickly zooming into the correct neighborhood. FAISS includes a highly optimized GPU implementation of HNSW. It is often the algorithm of choice for high-recall, low-latency applications, though it can be memory-intensive compared to methods like IVF-PQ.

Product Quantization (PQ) is a compression technique used in similarity search to dramatically reduce the memory footprint of high-dimensional vectors. It works by:

1. Splitting the original vector into several sub-vectors.
2. Quantizing each sub-vector separately using a small codebook (e.g., 256 centroids).
3. Representing the original vector by a short code composed of the indices of the nearest centroids for each segment. This allows billions of vectors to be stored in RAM. In FAISS, PQ is often combined with a coarse quantizer like Inverted File (IVF) to create the IVF-PQ index, which enables fast search over compressed data. The trade-off is a slight loss in distance calculation accuracy.

A unified embedding space is a joint vector representation where embeddings from different modalities (e.g., text, image, audio) are directly comparable using a similarity metric like cosine distance. This is the foundational data that FAISS indexes. Creating this space involves:

• Training multimodal models (e.g., CLIP, ImageBind) that align different data types into a shared semantic space.
• Using the model to encode diverse data into vectors of the same dimensionality. Once encoded, FAISS can perform cross-modal retrieval, such as finding relevant images for a text query, because the vectors inhabit the same mathematical space. The quality of the embedding model dictates the quality of the search results.

Hybrid search is an information retrieval technique that combines the results of two or more search methods, typically keyword-based (lexical) search and vector-based (semantic) search, to improve overall recall and precision. While FAISS excels at semantic search, it lacks native keyword filtering. A hybrid architecture might:

1. Use a traditional search engine (e.g., Elasticsearch) for exact keyword matches, filters, and range queries.
2. Use FAISS for finding semantically similar vectors.
3. Fuse the results using a scoring function (e.g., weighted sum, reciprocal rank fusion). This approach leverages the strengths of both methods: the precision of keywords for known entities and the recall of vector search for conceptual similarity.