Memory Vector Search

The embedding model is a neural network (e.g., a transformer) that converts raw data—text, images, audio—into high-dimensional numerical vectors called embeddings. This process, known as encoding, maps semantically similar items to nearby points in the vector space. The choice of model (e.g., OpenAI's text-embedding-3-small, Cohere's embed-english-v3.0) directly determines the quality of semantic search.

• Function: Transforms unstructured data into a searchable mathematical representation.
• Output: A dense vector (e.g., 768, 1536 dimensions) where geometric distance corresponds to semantic similarity.

A vector index is a data structure optimized for fast Approximate Nearest Neighbor (ANN) search. Exhaustively comparing a query vector against every stored vector is computationally prohibitive (O(N)). ANN indexes trade perfect accuracy for massive speed gains (sub-millisecond latency at scale).

Common index types include:

• HNSW (Hierarchical Navigable Small World): A graph-based method offering high recall and speed, used by Weaviate and Pinecone.
• IVF (Inverted File Index): Clusters vectors into Voronoi cells, searching only the most promising clusters, used by FAISS.
• PQ (Product Quantization): Compresses vectors to reduce memory footprint and accelerate distance calculations.

The distance metric is a mathematical function that quantifies the similarity between two vectors in the embedding space. The choice of metric must align with the properties of the embeddings produced by the model.

Key metrics include:

• Cosine Similarity: Measures the cosine of the angle between vectors. Most common for text embeddings, as it focuses on orientation rather than magnitude.
• Euclidean Distance (L2): Measures the straight-line distance between points. Often used for image and multimodal embeddings.
• Inner Product (Dot Product): Related to cosine similarity but affected by vector magnitude. Requires normalized vectors for consistent results.

The search returns vectors with the smallest distance (or largest similarity) to the query.

This component manages the live search operation. The query encoder uses the same embedding model to transform the agent's current context or question into a query vector. The retrieval logic then:

1. Executes the ANN search using the chosen distance metric.
2. Applies optional metadata filters (e.g., date > 2024, source = 'internal_docs') for hybrid search.
3. Ranks and returns the top-k most similar vectors (e.g., k=10).
4. May perform re-ranking using a cross-encoder model for higher precision on the initial results.

The raw data stored and retrieved is not just the vector. Each memory entry typically consists of:

• Chunked Content: The original text or data, segmented (chunked) into logical pieces (e.g., paragraphs, sections) before embedding to optimize for retrieval granularity.
• Vector Embedding: The numerical representation of that chunk.
• Metadata: Structured fields (e.g., source_id, timestamp, author, type) attached to the chunk. This enables filtered vector search, where semantic search is scoped to a relevant subset of memories.

This structure allows the system to return the relevant original text to the agent after a vector lookup.

The vector store is the persistent storage and serving layer that brings all components together. It is a specialized database (e.g., Pinecone, Weaviate, Qdrant, pgvector) that:

• Ingests and stores vector embeddings, their associated chunks, and metadata.
• Creates and maintains the ANN index on the vectors.
• Exposes an API for performing low-latency vector searches, often with filtering capabilities.
• Manages scalability through sharding and replication for large-scale agent deployments.

An Approximate Nearest Neighbor (ANN) index is a data structure that accelerates vector similarity search by trading off exact precision for dramatic gains in query speed and reduced memory usage. Instead of comparing a query vector against every stored vector (a brute-force O(n) operation), ANN algorithms organize vectors to enable sub-linear search times.

• Common Algorithms: Hierarchical Navigable Small Worlds (HNSW), Inverted File (IVF) indexes, and Locality-Sensitive Hashing (LSH).
• Trade-off: Controlled by parameters that balance recall (percentage of true nearest neighbors found) against query latency and index build time.
• Implementation: Core to vector databases like Pinecone, Weaviate, and Qdrant, and libraries like FAISS and Annoy.

An embedding model is a neural network, often a transformer, that encodes unstructured data (text, images, audio) into high-dimensional vector representations (embeddings). The quality of Memory Vector Search is fundamentally dependent on the embedding model's ability to map semantically similar items to nearby points in the vector space.

• Key Property: It creates a dense vector where geometric distance (e.g., cosine similarity) corresponds to semantic relatedness.
• Examples: OpenAI's text-embedding-3 models, Cohere's Embed models, and open-source models like BGE and E5.
• Fine-tuning: Domain-specific fine-tuning of embedding models is often required to align the semantic space with an agent's specialized knowledge base.

Hybrid Search is a retrieval strategy that combines the results of dense vector search (semantic) and sparse vector search (keyword-based, like BM25) to improve overall recall and precision. It addresses the limitations of pure semantic search, such as missing exact keyword matches or handling rare entities.

• Mechanism: Executes both search types in parallel and uses a fusion algorithm (e.g., reciprocal rank fusion) to merge and re-rank the combined result set.
• Metadata Filtering: Often combined with pre- or post-filtering on structured metadata (e.g., date > 2024, author = 'system') for precise scoping.
• Benefit: Provides a robust, general-purpose retrieval system for agent memory, catching both semantic intent and literal keyword references.

Recall and Precision are the primary evaluation metrics for Memory Vector Search quality, defining the trade-off central to ANN index configuration.

• Recall: The fraction of the true nearest neighbors (from a brute-force search) that are successfully retrieved by the ANN search. High recall is critical for agents that must not miss relevant context.
• Precision: The fraction of retrieved items that are actually part of the true nearest neighbors. High precision improves the signal-to-noise ratio for the agent's context window.
• Trade-off: Increasing an ANN index's search parameters (e.g., ef in HNSW) typically improves recall at the cost of higher latency and compute. Tuning this is a key engineering task.

A Vector Database is a specialized database system designed for the efficient storage, indexing, and retrieval of vector embeddings. It is the persistent storage backend for most production-scale agentic memory systems.

• Core Features: Native support for ANN indexes, metadata filtering, and real-time upserts. Often includes built-in embedding generation pipelines.
• Distinction from Vector Index Libraries: Provides database features like durability, replication, access control, and a query language, whereas libraries like FAISS are primarily in-memory indexes.
• Examples: Pinecone (managed), Weaviate (open-source), Qdrant (open-source), and pgvector (PostgreSQL extension).

A Similarity Metric (or distance function) is a mathematical function that quantifies the similarity or dissimilarity between two vectors in the embedding space. The choice of metric determines how "closeness" is defined for search.

• Cosine Similarity: Measures the cosine of the angle between two vectors. Most common for text embeddings, as it is magnitude-invariant and focuses on orientation.
• Inner Product (Dot Product): Related to cosine similarity but affected by vector magnitude. Used with normalized embeddings.
• Euclidean Distance (L2): Measures the straight-line distance between vectors. Common in computer vision embeddings.
• Index Alignment: The ANN index must be built and queried using the same metric the embeddings were optimized for.