Semantic Search

The foundational process of converting text, images, or other data into high-dimensional numerical vectors (embeddings) using transformer models. These vectors capture semantic meaning, positioning conceptually similar items close together in vector space.

• Embedding models like text-embedding-3-large or bge-large-en produce dense vectors of 1024–3072 dimensions
• Cosine similarity measures the angular distance between query and document vectors
• Semantic proximity enables retrieval of relevant documents even with zero keyword overlap
• Embeddings are stored in specialized vector databases for low-latency approximate nearest neighbor (ANN) search

Semantic search systems leverage Natural Language Understanding (NLU) to parse the user's true intent rather than treating the query as a bag of words. This resolves ambiguity, handles synonyms, and interprets complex phrasings.

• Distinguishes between "apple" the fruit vs. "Apple" the company using entity recognition
• Maps "how to fix a leaky faucet" to plumbing repair concepts without exact term matching
• Uses contextual query expansion to incorporate related terms from conversation history
• Handles multi-turn reasoning by maintaining dialogue state across sequential queries

Production semantic search systems combine sparse retrieval (lexical matching like BM25) with dense retrieval (vector similarity) to maximize both precision and recall. This dual approach ensures robust performance across all query types.

• BM25 excels at exact term matching for rare keywords, product codes, or IDs
• Dense vectors capture conceptual similarity for natural language queries
• Fusion algorithms like Reciprocal Rank Fusion (RRF) merge results from both indexes
• Conversational reranking applies a cross-encoder to reorder results based on full dialogue context

Semantic search is enhanced by knowledge graphs that provide structured, deterministic relationships between entities. This grounding layer ensures factual accuracy and enables reasoning across connected concepts.

• Entity resolution links query mentions to canonical entities in the graph
• Relationship traversal retrieves connected facts (e.g., "CEO of → founded by")
• Knowledge graph embeddings project graph structure into vector space for unified retrieval
• Reduces hallucination risk by anchoring responses to verified entity relationships

After initial retrieval, a cross-encoder model performs fine-grained relevance assessment by jointly encoding the query and each candidate document. This second-pass scoring dramatically improves result quality.

• Cross-encoders evaluate full query-document interaction, unlike bi-encoders
• Models like Cohere Rerank or bge-reranker-v2 reorder top-k candidates
• Multi-vector scoring considers the entire conversational context, not just the current query
• Ensures the most contextually appropriate document surfaces at the top position

Modern semantic search pipelines deliver results with minimal perceived latency through streaming inference and optimized serving infrastructure. This is critical for conversational interfaces where users expect instant responses.

• Token streaming delivers partial LLM outputs as they are generated
• Time-to-First-Token (TTFT) is a key metric measuring initial response delay
• Continuous batching maximizes GPU utilization during concurrent queries
• Edge deployment of embedding models reduces network round-trip latency for local applications

The mathematical foundation of semantic search. Vector embeddings are dense, high-dimensional numerical representations of words, sentences, or documents that capture semantic meaning as geometric proximity.

• Dimensionality: Typically 768 (BERT), 1536 (OpenAI), or 1024 dimensions
• Key property: Similar concepts cluster together in vector space regardless of keyword overlap
• Cosine similarity measures the angle between vectors to determine relevance
• Generated by encoder models trained on massive text corpora

Example: The vectors for 'automobile' and 'car' are close neighbors, while 'automobile' and 'banana' are distant—even though no keywords match.

A retrieval methodology that combines sparse lexical retrieval (BM25) with dense vector search to capture both exact keyword matches and conceptual meaning.

• BM25 component: Excels at matching rare, specific terms like product codes or technical jargon
• Dense component: Captures paraphrased queries and conceptual relationships
• Fusion algorithms: Reciprocal Rank Fusion (RRF) or weighted score combination merges results
• Outperforms either method alone on most enterprise benchmarks

Critical for domains where precision on unique identifiers must coexist with natural language understanding.

The algorithmic backbone enabling fast vector similarity search at scale. ANN algorithms trade a small amount of accuracy for dramatic speed improvements over exact k-nearest neighbor search.

• HNSW (Hierarchical Navigable Small World): Graph-based index with logarithmic search complexity
• IVF (Inverted File Index): Clusters vectors and searches only relevant partitions
• PQ (Product Quantization): Compresses vectors to reduce memory footprint
• DiskANN: Designed for billion-scale datasets that exceed RAM capacity

Without ANN, semantic search over millions of documents would be computationally infeasible for real-time applications.

A two-stage retrieval architecture where a lightweight bi-encoder performs initial candidate retrieval, followed by a more powerful cross-encoder that jointly processes the query and each document.

• Bi-encoder: Encodes query and documents independently for fast cosine similarity scoring
• Cross-encoder: Feeds query-document pairs through a transformer for deep relevance assessment
• Computational cost: Cross-encoders are 100-1000x slower but significantly more accurate
• Applied only to top-k candidates (typically 100-1000) from the first stage

This architecture delivers the speed of vector search with the accuracy of full attention-based comparison.

The methodology of segmenting documents into granular, semantically coherent units for embedding and retrieval. Chunking quality directly determines whether the right information fits within an LLM's context window.

• Fixed-size chunking: Splits by token count (e.g., 512 tokens) with overlap—simple but can break mid-sentence
• Semantic chunking: Uses sentence boundaries and embedding similarity to create coherent segments
• Hierarchical chunking: Maintains parent-child relationships between document sections and subsections
• Small-to-big retrieval: Retrieves small chunks but expands to surrounding context for LLM generation

Poor chunking is the most common cause of retrieval failure in production RAG systems.

The dominant training paradigm for modern embedding models. Contrastive learning trains encoders to pull semantically similar text pairs together in vector space while pushing dissimilar pairs apart.

• Positive pairs: Query-relevant document, paraphrased sentences, or question-answer pairs
• Negative pairs: Random documents or hard negatives (similar but irrelevant text)
• InfoNCE loss: The standard objective function maximizing mutual information between positive pairs
• Hard negative mining: Strategically selecting challenging negatives improves embedding quality

Models like E5, BGE, and GTE use contrastive objectives to achieve state-of-the-art retrieval performance on benchmarks like MTEB.