Chunk Indexing

These are purpose-built databases designed to store, index, and query high-dimensional vector embeddings at scale. They are the primary infrastructure for chunk indexing in production RAG systems.

Key features include:

• Approximate Nearest Neighbor (ANN) Search: Algorithms like HNSW or IVF that enable fast similarity search across billions of vectors.
• Metadata Filtering: The ability to combine vector similarity search with exact filters on chunk metadata (e.g., source = 'Q3_report.pdf').
• Hybrid Search Support: Native integration of sparse (keyword/BM25) and dense (vector) search for improved recall.

Examples: Pinecone, Weaviate, Qdrant, Milvus, Vespa.

A data framework specifically designed for building LLM-powered applications with a strong focus on ingestion, indexing, and retrieval. Its Node object is the fundamental chunk unit.

Core Indexing Components:

• Node Parsers: Convert documents into Node objects (chunks) with configurable strategies (semantic, hierarchical).
• Vector Store Index: The primary index type that creates and stores vector embeddings for each node.
• Metadata Extractors: Automatically pull metadata (titles, dates, entities) from chunks during indexing.
• Index Composability: Allows building complex indices like hierarchical or keyword-tables alongside vector indices.

A framework for developing applications with LLMs that provides modular components and chains. Its indexing utilities are often used to integrate with external vector databases.

Key Indexing Abstractions:

• Document Loaders & Text Splitters: Ingest and chunk documents before indexing.
• VectorStore Interface: A unified API for interacting with different vector databases (e.g., from_documents() method).
• Retrievers: Configurable objects that define the retrieval logic (similarity search, MMR, self-query) on top of an indexed store.
• Indexing API: Tools for incremental indexing and managing large document collections.

Standalone libraries that implement Approximate Nearest Neighbor (ANN) algorithms. These can be embedded directly into applications or used to build custom vector indexing layers.

Common Algorithms & Libraries:

• HNSW (Hierarchical Navigable Small World): A graph-based algorithm offering a strong trade-off between speed, accuracy, and memory. Implemented in FAISS and hnswlib.
• IVF (Inverted File Index): A clustering-based algorithm that is highly memory-efficient. Core to FAISS.
• SCANN (Scalable Nearest Neighbors): Google's library for maximum recall at high speed.
• FAISS (Facebook AI Similarity Search): The most widely used library, providing GPU acceleration and various index types.

A vector database is a specialized storage system optimized for indexing and querying high-dimensional vector embeddings. It is the primary infrastructure where chunk indexes are stored, enabling fast approximate nearest neighbor (ANN) search. Key features include:

• Dense Indexing: Stores the vector representation of each chunk.
• Metadata Filtering: Allows hybrid queries combining semantic search with filters on chunk attributes (e.g., source, date).
• Scalability: Designed to handle billions of vectors with low-latency retrieval. Examples include Pinecone, Weaviate, and Qdrant.

An embedding model is a neural network that converts a text chunk into a fixed-length, dense numerical vector (an embedding). This model is executed during the indexing phase to create the searchable representations stored in the vector database. Critical characteristics include:

• Dimensionality: Output vector size (e.g., 384, 768, 1536 dimensions).
• Domain Adaptation: Models can be general-purpose (e.g., OpenAI's text-embedding-ada-002) or fine-tuned on specific corpora.
• Semantic Preservation: The cosine distance between two chunk embeddings should correlate with their semantic similarity. The choice of embedding model fundamentally determines the quality of the semantic index.

Approximate Nearest Neighbor (ANN) search is a family of algorithms that efficiently find the vectors closest to a query vector in high-dimensional space, trading perfect accuracy for massive speed gains. It is the computational core of querying a chunk index. Popular algorithms include:

• HNSW (Hierarchical Navigable Small World): A graph-based method offering high recall and speed.
• IVF (Inverted File Index): Clusters vectors into Voronoi cells for coarse-to-fine search.
• PQ (Product Quantization): Compresses vectors to reduce memory footprint and accelerate distance calculations. These algorithms make querying billion-scale chunk indexes feasible with millisecond latency.

Hybrid search is a retrieval methodology that combines the results of dense vector search (semantic) and sparse lexical search (keyword-based) to improve overall recall and precision. In chunk indexing, it leverages both the vector embedding and the raw text/metadata of chunks.

• Dense Vector Search: Finds semantically similar chunks using ANN search on embeddings.
• Sparse Lexical Search: Uses traditional text search (e.g., BM25) on the chunk's raw text. Results from both methods are fused using techniques like reciprocal rank fusion (RRF) to produce a final ranked list, making the chunk index robust to varied query types.