Vector Database

Vector databases are purpose-built to store and manage dense vector embeddings—numerical representations of legal documents, clauses, or concepts in high-dimensional space (typically 768 to 4096 dimensions). Unlike traditional databases that index scalar values, these systems organize data as points in a continuous vector space where proximity equals semantic similarity. Each embedding is stored alongside its original metadata (case citation, contract ID, jurisdiction) and payload, enabling hybrid filtering. The storage engine must handle billions of vectors efficiently, often using compressed formats like Product Quantization (PQ) to reduce memory footprint from 4KB per vector to as little as 64 bytes while preserving recall quality for legal retrieval tasks.

Exact nearest neighbor search over millions of legal embeddings is computationally prohibitive. Vector databases employ Approximate Nearest Neighbor (ANN) algorithms that trade marginal accuracy for orders-of-magnitude speed improvements. The most common index structure is Hierarchical Navigable Small World (HNSW)—a multi-layered graph where each layer skips increasingly larger distances. Search begins at the topmost sparse layer and descends through denser local graphs, achieving logarithmic complexity with recall rates exceeding 95%. Alternative indexing strategies include Inverted File Index (IVF) for disk-based storage and DiskANN for billion-scale collections that exceed available RAM, critical for large legal document corpora.

Vector databases compute relevance using mathematical distance functions that quantify how close two embeddings are in vector space. The choice of metric fundamentally shapes retrieval behavior:

• Cosine Similarity: Measures the angle between vectors, ignoring magnitude. Dominant in legal NLP because it normalizes for document length, preventing longer contracts from dominating results.
• Euclidean Distance (L2): Measures straight-line distance. Sensitive to embedding magnitude, useful when vector norms carry semantic weight.
• Dot Product: Equivalent to cosine similarity when vectors are normalized. Preferred for maximum inner product search in recommendation-style retrieval.
• Manhattan Distance (L1): Sum of absolute differences. Used in sparse or binary embedding contexts. The database must support index-time metric selection, as changing the distance function typically requires rebuilding the entire ANN index.

Pure semantic search is insufficient for legal applications that require precise scoping constraints. Vector databases support pre-filtering (applying metadata constraints before vector search) and post-filtering (filtering ANN results by metadata). Pre-filtering ensures exact match guarantees—for example, restricting search to contracts governed by Delaware law or cases within a specific date range—but can degrade ANN recall if the filtered set is small. Advanced implementations use filterable HNSW or range search that integrates boolean predicates directly into graph traversal. This enables queries like: 'Find clauses semantically similar to this indemnification provision, but only in agreements executed after 2020.'

Legal document collections are dynamic—new cases are published, contracts are amended, and regulations evolve. Vector databases must support real-time insertions, updates, and deletions without requiring full index rebuilds. HNSW graphs support incremental insertion by traversing the existing graph to find the new vector's neighbors and adding edges, though this can degrade search performance over time as the graph structure becomes suboptimal. Deletions are typically handled via tombstone markers rather than immediate graph restructuring. For high-throughput legal pipelines, some databases offer out-of-place indexing where a new index is built asynchronously and swapped atomically, ensuring query performance remains unaffected during bulk ingestion of newly embedded case law.

When a single node cannot hold the entire legal embedding corpus, vector databases employ horizontal sharding—partitioning vectors across multiple machines. Sharding strategies include:

• Random sharding: Uniformly distributes vectors, ensuring balanced load but requiring every shard to be queried (scatter-gather).
• Semantic sharding: Clusters vectors by similarity so queries route to only relevant shards, reducing network overhead but risking hotspots.
• Metadata-based sharding: Partitions by attributes like jurisdiction or practice area, aligning with natural legal taxonomies. Distributed deployments must also handle consensus protocols for index consistency and query fan-out where a single search fans out to multiple shards and results are merged via Reciprocal Rank Fusion (RRF) before returning to the client.

A class of algorithms that trade marginal accuracy for substantial speed improvements when searching high-dimensional vector spaces. In legal retrieval, ANN enables sub-100ms query latency across millions of case law embeddings by avoiding exhaustive distance calculations. Key tradeoff: recall vs. queries per second. Common implementations include HNSW, IVF-PQ, and DiskANN.

A graph-based ANN index that builds a multi-layered navigable structure to achieve logarithmic search complexity. How it works:

• Upper layers contain long-range edges for coarse traversal
• Lower layers refine with short-range connections
• Greedy search descends through layers, maintaining a dynamic candidate set HNSW delivers state-of-the-art recall-speed tradeoffs and is the default index in many production vector databases.

A vector compression technique that decomposes high-dimensional embeddings into smaller sub-vectors and quantizes each independently using k-means clustering. Memory reduction: a 1024-dimensional float32 vector (4KB) can be compressed to 64 bytes with PQ, enabling billion-scale legal document indices to fit in RAM. The tradeoff is a slight recall degradation that can be mitigated by re-ranking top candidates with exact distance computation.

A training method that produces embedding vectors whose truncated prefixes remain useful for similarity search. A single 1024-dimensional embedding can be sliced to 256, 512, or 768 dimensions without retraining. Legal use case: store full-dimensional embeddings for high-precision case law search while using truncated versions for fast, low-cost pre-filtering—enabling flexible dimensionality trade-offs within a single index.

An algorithm that combines ranked result lists from multiple retrieval systems by computing a reciprocal rank score: score = 1 / (k + rank), where k is a constant (typically 60). Why it matters: RRF enables seamless fusion of sparse BM25 lexical results with dense vector search results without requiring score calibration. In legal hybrid search, this ensures both exact keyword matches and semantically relevant documents appear in the final ranked list.