Semantic Cache

The core mechanism that enables semantic matching. When a query is received, it is first converted into a high-dimensional vector embedding using a model like OpenAI's text-embedding-3-small or an open-source alternative. This numerical representation captures the semantic meaning of the query, allowing the system to compare the intent of two different phrasings.

• Process: Query Text → Embedding Model → Vector Representation
• Key Property: Semantically similar queries (e.g., "Explain caching" and "What is a cache?") produce vectors that are close together in the vector space, as measured by cosine similarity.

Once a query is embedded, the system performs a nearest neighbor search within a vector database (e.g., Pinecone, Weaviate, pgvector) to find previously cached queries with similar meanings. A similarity threshold (e.g., cosine similarity > 0.85) determines a cache hit.

• Cache Hit: If a cached query's embedding is within the threshold, the associated cached result is returned.
• Cache Miss: If no similar embedding is found, the query is processed normally (e.g., sent to an LLM or API), and the new query/result pair is embedded and stored for future use.
• Critical Tuning: The threshold balances recall (finding relevant matches) with precision (avoiding incorrect matches).

Semantic caches must distinguish between cacheable and non-cacheable operations.

• Deterministic Caching: Ideal for pure functions or APIs where the same semantic input always produces the same output (e.g., data transformation, fixed database lookup). Cache invalidation is straightforward.
• Non-Deterministic Caching: Challenging for LLM calls where the same prompt can produce varied outputs (e.g., creative writing, non-temperature-zero completions). Strategies include:
  • Caching only with temperature=0.
  • Storing multiple possible outputs or a canonical "best" answer.
  • Using more conservative similarity thresholds to avoid inappropriate reuse.

The cache key in a semantic cache is a composite object, not a simple string. It typically includes:

1. The query's embedding vector (for similarity search).
2. A namespace or session ID to isolate caches per user, tenant, or agent context.
3. Model identifiers (e.g., gpt-4, claude-3-opus) as results are model-specific.
4. System prompt or function signature to differentiate intent (e.g., a query to a "coding assistant" vs. a "customer support" agent).

This prevents a query like "How do I fix this?" from returning a cached coding answer when the user is now asking about a broken appliance.

In an agentic workflow, semantic caching intercepts calls before they reach external services.

• Workflow: Agent generates a tool call request (e.g., get_weather(location="Paris")) → Request is semantically compared to cached tool calls → On a hit, the cached API response is returned instantly.
• Benefit: Eliminates redundant, costly, or rate-limited API calls for semantically identical operations (e.g., multiple agents asking for the same stock price within a short window).
• Schema Awareness: Can be integrated with API Schema Integration to understand and cache based on the normalized intent of a call, not just its surface parameters.

Semantic caches require specialized policies to manage staleness.

• Time-To-Live (TTL): A base policy to expire all entries after a duration (e.g., 1 hour for weather data, 24 hours for static facts).
• Event-Driven Invalidation: Listening to webhooks or database change events to purge caches related to specific data entities.
• Embedding Cluster Invalidation: When a core piece of data changes, invalidating all cached queries whose embeddings fall within a related cluster in vector space.
• Statistical Eviction: Using metrics like cache hit ratio and latency savings to prioritize retaining high-value semantic entries, often employing policies like LRU or LFU on the embedding space.

A deterministic cache stores the results of pure functions, where the same input arguments always produce an identical output. This guarantees a valid cache hit, as the output is solely derived from the input with no external dependencies. It is a foundational concept for any caching system that relies on exact key matching.

• Key Principle: Idempotency and referential transparency.
• Contrast with Semantic Cache: A deterministic cache requires an exact match on the input (e.g., a hash of the function arguments), whereas a semantic cache matches on similar meaning.

The cache hit ratio is the primary performance metric for any caching system, calculated as (Number of Cache Hits / Total Cache Requests) * 100. A high ratio indicates effective cache utilization and reduced load on primary data sources (like LLM APIs or databases).

• Semantic Cache Impact: By matching on intent, a semantic cache can achieve a higher effective hit ratio than a deterministic cache for natural language queries, as varied phrasings of the same question will result in hits.
• Monitoring: This metric is crucial for capacity planning and tuning cache eviction policies.

A vector database is the specialized storage system that enables semantic caching. It indexes high-dimensional embeddings (numerical representations of semantic meaning) for rapid Approximate Nearest Neighbor (ANN) search.

• Core Function: When a new query arrives, its embedding is generated and compared against cached query embeddings in the vector store to find semantically similar past queries.
• Key Components: Includes ANN algorithms (HNSW, IVF), metadata filtering, and hybrid search capabilities. It is the enabling technology that makes semantic similarity matching computationally feasible.

Cache eviction is the process of removing items from a full cache. The Least Recently Used (LRU) policy evicts the item that hasn't been accessed for the longest time.

• Semantic Cache Nuance: Eviction must consider the embedding space. Simply evicting by recency may discard a unique semantic intent. Advanced systems may use LRU-K or Adaptive Replacement Cache (ARC) to balance recency and frequency of semantic access patterns.
• Admission Policies: Deciding what to cache is as important as what to evict. A semantic cache may use a utility score based on compute cost and query frequency to determine admission.

In a read-through cache, the cache sits logically between the application and the data source. On a cache miss, the cache itself is responsible for fetching the data from the primary source (e.g., an LLM API), storing it, and then returning it.

• Architectural Fit: This pattern is well-suited for semantic caching agents. The caching layer can transparently handle the complexity of embedding generation, similarity search, and LLM fallback without the application code managing the logic.
• Benefit: Simplifies application code and centralizes cache management logic.

Inference optimization encompasses techniques to reduce the cost and latency of running machine learning models. Semantic caching is a direct, high-impact optimization for LLM inference.

• Cost Reduction: Avoids redundant, expensive LLM API calls for semantically identical requests.
• Latency Reduction: Returns a cached result in milliseconds vs. seconds for a full model inference.
• Synergy with Other Techniques: Works alongside KV caching (which optimizes within a single generation) and continuous batching (which optimizes across multiple requests).