Semantic Cache

Instead of exact key matching, the cache indexes queries using vector embeddings generated by an encoder model. When a new query arrives, it is converted into the same vector space. The cache then performs an Approximate Nearest Neighbor (ANN) search to find stored queries with high cosine similarity.

• Uses distance metrics like cosine similarity or Euclidean distance
• A similarity threshold (e.g., 0.95) determines a cache hit
• Eliminates cache misses caused by minor phrasing differences like 'What is semantic caching?' vs 'Define semantic cache'

A tunable parameter defines the minimum similarity score required to serve a cached response. This directly controls the trade-off between cache hit rate and semantic precision.

• A high threshold (0.98) ensures near-identical meaning but yields fewer hits
• A lower threshold (0.85) increases hit rates but risks serving slightly off-topic responses
• Thresholds can be dynamically adjusted per use case or tenant in sovereign deployments

Every cached entry carries a Time-To-Live (TTL) that dictates how long it remains valid. This prevents serving stale factual data in rapidly changing domains.

• Absolute TTLs expire entries after a fixed duration (e.g., 3600 seconds)
• Sliding TTLs reset the timer on each cache hit, keeping hot data alive
• Integration with event-driven invalidation purges entries when upstream data sources change

When the cache reaches memory capacity, an eviction policy deterministically selects which entries to discard. The choice of policy directly impacts hit rate under constrained resources.

• Least Recently Used (LRU): Evicts the entry with the oldest access timestamp
• Least Frequently Used (LFU): Evicts entries with the lowest access count over a time window
• Cost-Aware Eviction: Prioritizes keeping entries that were computationally expensive to generate, such as long-context LLM responses

To avoid brute-force similarity comparisons over millions of embeddings, semantic caches rely on specialized vector index structures that enable sub-linear search complexity.

• Hierarchical Navigable Small World (HNSW) graphs provide logarithmic search complexity
• Locality-Sensitive Hashing (LSH) buckets similar vectors together for rapid retrieval
• Product Quantization (PQ) compresses vectors to drastically reduce memory footprint while preserving approximate distance calculations

In enterprise deployments, the cache must enforce strict tenant isolation to prevent cross-organization data leakage through the caching layer.

• Logical isolation via namespace partitioning per tenant ID
• Physical isolation through dedicated cache clusters for high-compliance tenants
• Encrypted at rest using tenant-specific keys, ensuring cached embeddings and responses remain confidential within jurisdictional boundaries

An algorithmic technique that hashes high-dimensional input vectors so that similar items map to the same buckets with high probability. LSH enables fast approximate nearest neighbor (ANN) search by trading a small amount of accuracy for massive speed gains.

• How it works: Projects embeddings via random hyperplanes; vectors landing on the same side share a bucket.
• Use case: Powers sub-millisecond lookups in billion-scale semantic caches.
• Trade-off: Controllable false negative rate vs. exhaustive KNN search.

A cascading failure scenario where a popular cache entry expires, causing a flood of concurrent requests to simultaneously hit the origin model or database. This thundering herd overwhelms backend resources, often triggering a circuit breaker.

• Mitigation: Probabilistic early expiration recomputes the value before expiry.
• Mitigation: Locking allows only one request to regenerate the value while others wait.
• Impact: Can cause latency spikes from < 10ms to multiple seconds.

A vector compression technique that decomposes high-dimensional embeddings into smaller sub-vectors and quantizes each independently. PQ drastically reduces the memory footprint of a semantic cache, enabling billion-scale indices in RAM.

• Mechanism: Splits a 768-dim vector into 96 sub-vectors of 8 dimensions; each sub-vector is mapped to the nearest centroid from a codebook.
• Compression: Reduces memory from 4 bytes/float to a few bits per dimension.
• Trade-off: Introduces a small reconstruction error, slightly reducing recall.

A distributed hashing scheme that minimizes key remapping when cache nodes are added or removed. Critical for maintaining high hit rates in elastic distributed cache layers where nodes scale dynamically.

• Ring Architecture: Nodes and keys are mapped onto a logical ring; each key is assigned to the nearest clockwise node.
• Resilience: Only K/N keys remap when a node fails, where K is total keys and N is node count.
• Virtual Nodes: Each physical node maps to multiple points on the ring to balance load distribution.