Embedding Cache

The cache intercepts text inputs before they reach the embedding model. A hash of the raw text (e.g., SHA-256) acts as the lookup key. On a cache hit, the pre-computed vector is returned in microseconds. On a miss, the text is sent to the encoder model, and the resulting vector is stored in the cache with an associated TTL (Time-to-Live). This is distinct from a semantic cache, which matches on meaning; an embedding cache requires an exact text match to guarantee deterministic vector identity.

Neural network inference is the dominant cost in vectorization. A single call to an encoder like text-embedding-3-large can take 10-50ms. A cache hit in Redis or Memcached typically takes < 1ms. This represents a 10x to 50x latency reduction per cached item. In a pipeline with a strict P99 latency budget, caching frequently accessed documents (e.g., a company's homepage or a popular knowledge base article) prevents the embedding step from consuming the entire time allocation.

The primary challenge is maintaining consistency between the source document and its cached vector. Common strategies include:

• TTL-Based Expiration: Set a time limit (e.g., 24 hours) after which the vector is re-computed. Simple but can serve stale data.
• Event-Driven Invalidation: A database trigger or webhook purges the cache entry immediately when the source document is updated. Ensures strong consistency.
• Versioned Keys: Append a document version number to the cache key. A new version automatically generates a cache miss, populating a new entry without deleting the old one.

The choice of backend depends on the scale and access pattern:

• In-Memory (Redis/Memcached): Sub-millisecond latency. Ideal for hot caches under 100GB. Volatile.
• Distributed (Apache Cassandra/DynamoDB): High availability and durability. Suitable for terabyte-scale caches where recomputation cost is prohibitive.
• Local Disk (LMDB/SQLite): Zero network latency. Useful for edge deployments or single-node development environments.
• Vector Databases: Some vector DBs (like Milvus or Qdrant) can serve as both the cache and the primary index, storing raw vectors without re-indexing.

Embedding API calls are metered by token count. Caching directly translates to reduced API spend. For a system processing 1 million queries per day with a 70% cache hit rate on embeddings, 700,000 inference calls are eliminated. At $0.00013 per 1K tokens (OpenAI Ada v2), this can save thousands of dollars monthly. Additionally, it reduces GPU utilization for self-hosted models, freeing compute for generation tasks.

An embedding cache is distinct from a KV-Cache. The embedding cache stores the final, fixed-length vector for an entire text block. The KV-Cache stores the intermediate key and value tensors for every token in a sequence during autoregressive generation. The embedding cache eliminates the encoding step before retrieval; the KV-Cache eliminates redundant computation during generation. Both are critical latency budgeting tools but operate at different stages of the RAG pipeline.

The cache hit ratio is the percentage of embedding requests successfully served from the cache rather than triggering a costly neural network inference. A high hit ratio directly correlates with lower P99 latency and reduced GPU compute costs. Monitoring this metric reveals the cache's effectiveness.

• Formula: (Cache Hits / Total Requests) × 100
• A ratio above 90% indicates strong temporal locality in your query patterns.
• A sudden drop often signals a shift in user behavior or a cache stampede event.

A semantic cache stores query-answer pairs based on vector similarity rather than exact string matching. When a new query arrives, its embedding is compared to cached embeddings. If a semantically similar query is found within a defined threshold, the cached result is returned, bypassing both embedding generation and LLM inference.

• Key Benefit: Handles paraphrased queries that a traditional key-value store would miss.
• Trade-off: Introduces a similarity search overhead that must be faster than the inference it replaces.

The eviction policy determines which embeddings are discarded when the cache reaches its memory limit. The choice of algorithm directly impacts the cache hit ratio under constrained resources.

• LRU (Least Recently Used): Discards the embedding that hasn't been accessed for the longest time. Optimal for recency-biased workloads.
• TTL (Time-to-Live): Expires entries after a fixed duration. Essential when source documents are updated frequently to prevent stale embeddings.
• LFU (Least Frequently Used): Tracks access frequency to retain popular, long-tail embeddings.

Cache warming is the proactive process of pre-computing and loading embeddings for a known set of high-value documents or frequent queries before a system goes live. This prevents the latency penalty of cold starts, where initial users would otherwise experience slow responses as the cache populates on demand.

• Common warming sources include nightly batch jobs on your document corpus.
• Critical for deployments following model updates where the old KV-Cache is invalidated.

P99 latency represents the maximum response time experienced by 99% of requests. While average latency might look healthy, a high P99 reveals that 1 in 100 users suffers a significantly slower experience—often due to a cache miss forcing a full embedding computation and vector search.

• Embedding caches are a primary tool for reducing tail latency.
• An SLO (Service Level Objective) is often defined around P99, such as 'P99 embedding retrieval < 5ms'.

A cache stampede is a cascading failure mode triggered when a highly popular cache entry expires. Multiple concurrent requests simultaneously discover the miss and flood the backend inference service with identical embedding generation requests, potentially causing an overload.

• Mitigation: Implement jitter on TTL expiration or use probabilistic early recomputation to refresh the hot entry before it expires.