Context Retrieval

A vector embedding is a dense, numerical representation of data (like text, images, or audio) in a high-dimensional space, where semantically similar items are positioned closer together. This transformation is performed by an embedding model (e.g., text-embedding-ada-002, BGE, E5).

• Purpose: Enables mathematical comparison (via similarity search) rather than brittle keyword matching.
• Output: A fixed-length array of floating-point numbers (e.g., 768 or 1536 dimensions).
• Key Property: The cosine similarity or dot product between two embedding vectors quantifies their semantic relatedness.

Semantic search is the retrieval of information based on the meaning of a query, not just lexical overlap. It works by comparing the vector embedding of a search query against a pre-computed index of document embeddings.

• Core Mechanism: Calculates similarity scores (e.g., cosine similarity) between the query vector and all candidate vectors in the index.
• Result: Returns a ranked list of the most semantically relevant documents or chunks.
• Contrast: Differs from traditional keyword search (e.g., BM25), which matches on term frequency and can miss paraphrases or conceptual links.

A retrieval index is a specialized data structure optimized for fast similarity search over high-dimensional vectors. It is the queried component of a vector database or search engine.

• Common Types:
  • Flat Index: Performs an exhaustive, accurate search (slow for large datasets).
  • Approximate Nearest Neighbor (ANN) Index: Uses algorithms like HNSW (Hierarchical Navigable Small World) or IVF (Inverted File Index) for fast, approximate search at scale.
• Function: Maps from a vector embedding back to the original content (the chunk) and its metadata.

Query formulation is the process of transforming a user's raw input or an agent's internal state into an effective search query for the retrieval system. Poor formulation is a primary cause of retrieval failure.

• Techniques:
  • Query Expansion: Adding synonyms or related terms.
  • Query Rewriting: Using a lightweight LLM to rephrase the query for clarity and specificity.
  • Hybrid Query: Combining a semantic vector search with a sparse keyword (BM25) search for improved recall.
• Agentic Context: An autonomous agent may generate a search query based on its current plan, previous actions, or perceived knowledge gaps.

Re-ranking is a secondary, often more computationally expensive, step that refines the results from an initial, fast retrieval pass. It improves precision by re-scoring a small candidate set.

• Purpose: To correct for limitations in the first-stage ANN index, which trades some accuracy for speed.
• Methods:
  • Cross-Encoder Models: A transformer that takes the query and a candidate chunk as a paired input to produce a more accurate relevance score.
  • LLM-as-Judge: Using a large language model to evaluate and rank retrieved passages based on instructions.
• Trade-off: Adds latency but significantly improves the quality of the final retrieved context.

Retrieval-Augmented Generation (RAG) is the overarching architecture that integrates context retrieval with a generative language model. It is the primary application pattern for context retrieval in agentic systems.

• Workflow:
  1. Retrieve: Fetch relevant context chunks based on the user query.
  2. Augment: Inject the retrieved context into the LLM's prompt.
  3. Generate: The LLM produces a final answer grounded in the provided evidence.
• Key Benefit: Mitigates hallucination by tethering the model to factual, external data.
• Agentic Use: Enables agents to access a persistent, evolving knowledge base beyond their static training data.

Hybrid search is a retrieval strategy that combines the strengths of semantic search (vector-based) and lexical search (keyword-based, e.g., BM25) to improve recall and precision.

• Rationale: Semantic search understands meaning but can miss exact term matches. Lexical search finds exact terms but misses synonyms. Hybrid covers both.
• Implementation: Runs both search types in parallel and uses a fusion algorithm (e.g., reciprocal rank fusion) to combine the ranked result lists into a single, superior list.
• Result: More resilient retrieval that handles a wider variety of query types, crucial for reliable context provisioning in agents.

Re-ranking is a post-processing step in retrieval where an initial set of candidate documents (from semantic or hybrid search) is re-scored by a more computationally intensive model to improve the final ranking.

• Purpose: The initial retrieval (ANN search) is fast but approximate. A re-ranker model evaluates the true relevance of each candidate to the query with higher accuracy.
• Model Types: Can be a cross-encoder model (which processes the query and document together) or a learned listwise ranking model.
• Impact: Drastically improves the quality of the top-ranked context passed to the LLM, leading to more accurate and relevant agent responses.