Semantic Cache

Instead of exact string matching, the cache transforms incoming queries into high-dimensional vector embeddings. It then performs an Approximate Nearest Neighbor (ANN) search against stored cache keys. A response is served if the cosine similarity or Euclidean distance falls within a defined threshold, capturing paraphrases and synonyms that a traditional key-value store would miss.

The core logic relies on a tunable similarity score (e.g., 0.95 cosine similarity). This threshold balances precision and recall:

• High Threshold (>0.98): Strict matching, minimizing false positives for fact-based queries.
• Lower Threshold (<0.90): Aggressive caching, useful for subjective or conversational prompts where approximate meaning is sufficient. This prevents serving incorrect answers to queries that are only superficially similar.

Semantic caches implement intelligent eviction beyond standard TTL (Time-To-Live) or LRU (Least Recently Used). They combine:

• Semantic TTL: Short lifespans for volatile topics (news) and longer ones for static knowledge (code syntax).
• Semantic Clustering Eviction: When the cache is full, it evicts entries from the densest semantic clusters first, preserving diversity in the cached knowledge base and preventing a single topic from dominating the storage.

Before caching, the system performs deterministic normalization on the query to strip noise without altering meaning. This includes lowercasing, removing stop words, and standardizing entities (e.g., 'USA' to 'United States'). This step ensures that minor typographical variations don't result in cache misses, while the semantic matching layer handles the heavier linguistic variations.

Advanced caches don't just embed the user's immediate query. They generate a composite cache key by concatenating the system prompt, recent conversation history, and the current query into a single block before embedding. This prevents the cache from returning a generic response when the specific conversational context (e.g., user role, prior topic) would require a fundamentally different answer.

The cache integrates with the LLM pipeline via two primary strategies:

• Read-Through: The application checks the cache first. On a miss, it queries the LLM and populates the cache asynchronously.
• Write-Through: The application writes the LLM response to the cache simultaneously with serving it to the user, ensuring immediate availability for subsequent near-duplicate requests. This architecture guarantees data consistency while optimizing for high-traffic scenarios.

A distinct caching mechanism that stores the Key and Value tensors of previously computed tokens in a transformer model. Unlike a semantic cache which stores final text responses, the KV-Cache stores intermediate computational state to accelerate autoregressive generation. When a user repeats a prompt prefix, the model reuses the stored tensors instead of recomputing them, drastically reducing time-to-first-token latency.

The technique of rewriting ambiguous queries into precise search strings. In a semantic cache pipeline, a query rewriter can normalize diverse user inputs into a canonical form before the cache lookup. For example, 'fix my login' and 'cannot sign in' are both rewritten to 'account access issue'. This increases the cache hit rate by reducing the variance the embedding model must handle.

The engineering practice of allocating strict time limits to each step of a retrieval pipeline. A semantic cache is the first line of defense in a latency budget. If a cache lookup exceeds 5-10ms, it fails to provide value over a standard LLM generation. This requires optimizing the ANN search and network calls to ensure the cache check is an order of magnitude faster than model inference.