Cache Key

A cache key must be deterministic: identical inputs must always generate the same key. This is foundational for a deterministic cache. The key is typically derived from a hash of the function signature and its arguments. For example, a tool call to get_weather(location="London", units="celsius") should always produce the same key, ensuring the same result is returned from the cache for the same logical request.

The key must be unique enough to distinguish between different logical requests. A collision (two different requests producing the same key) leads to incorrect data being served. Effective strategies include:

• Hashing the full normalized request (function name, sorted parameters).
• Including a namespace or tenant identifier for multi-user systems.
• Incorporating the version of the API schema or data model. For instance, get_user(123) and get_company(123) must not collide, which can be prevented by including the function name in the hash.

This characteristic defines the cache's flexibility. A syntactic key uses an exact string match of the request. A semantic cache key is designed for semantic caching, where the key is based on the meaning of the request.

• Syntactic Example: Key = hash("temp in NYC"). Only that exact phrase yields a hit.
• Semantic Example: Key = hash(embedding_vector("NYC temperature")). Phrases like "weather in New York" may yield a hit if their embeddings are similar, greatly increasing hit rates for LLM interactions.

In agentic workflows, a cache key often must incorporate session context beyond the immediate API call parameters to maintain correctness. This includes:

• User identity and permission scopes (a user should not see another user's cached data).
• Conversation thread ID or session token.
• Model configuration (e.g., GPT-4 vs. Claude-3).
• External state like current data freshness requirements. Omitting context leads to security violations or stale data being served across sessions.

The granularity of a cache key dictates the scope of what is cached. It's a trade-off between reuse and precision.

• Coarse-grained key: get_dashboard(). Caches the entire dashboard output. High reuse, but invalidated by any component change.
• Fine-grained key: get_chart(dataset="Q1_sales", metric="revenue"). Caches individual components. Enables cache partitioning and more precise cache invalidation but may lower the hit ratio if requests are highly variable.

The process of generating the key must be fast and produce a serializable string. Performance is critical as key generation happens on every cache lookup.

• Use fast, cryptographic hash functions (e.g., MD5, SHA-1) for uniqueness.
• Serialize complex objects (like parameter dictionaries) in a consistent, canonical order (e.g., sorted JSON).
• Avoid including extremely large objects (like full images) in the key; instead, hash them or use a content fingerprint. Inefficient key creation can negate the performance benefits of caching.

These are the two fundamental outcomes of a cache lookup using a cache key.

• Cache Hit: The requested key is found in the cache, allowing data to be served with low latency. A high hit ratio is the primary goal of caching.
• Cache Miss: The key is not found, forcing a slower retrieval from the primary data source (e.g., database, API). The retrieved data is then stored in the cache using the generated key for future requests. The effectiveness of a cache key's design directly impacts the frequency of hits versus misses.

When a cache is full, these algorithms decide which item to remove. The cache key is the handle for these operations.

• Least Recently Used (LRU): Evicts the key that has not been accessed for the longest time. It assumes recently used keys will be used again.
• Least Frequently Used (LFU): Evicts the key with the lowest number of accesses over a period. It assumes frequently used keys will remain useful. Advanced variants like LRU-K and Adaptive Replacement Cache (ARC) combine recency and frequency for more adaptive performance.

This defines how updates to the primary data source are reflected in cached values associated with their keys.

• Strong Consistency: Any read following a write returns the updated value. This often requires synchronous invalidation of the cache key, impacting performance.
• Eventual Consistency: The cache is updated asynchronously. Reads may temporarily return stale data for a key, but the system guarantees all copies will converge to the same state. This model favors availability and low latency. The choice dictates the cache invalidation strategy for a given key.

These are architectural patterns defining how an application interacts with the cache using keys.

• Cache-Aside (Lazy Loading): The application code directly manages the cache. On a miss, it fetches data, computes the key, and populates the cache. This offers maximum control but couples logic to the app.
• Read-Through Cache: The cache provider itself handles misses. The application requests data by key; if absent, the cache loads it from the configured source. This provides a cleaner abstraction for the application layer.

A specialized cache where the cache key is derived from the semantic meaning or intent of a query, rather than an exact string match. This is critical for caching outputs from large language models or complex computations.

• For example, queries like "What's the weather in Paris?" and "Get me the Paris forecast" would generate similar semantic keys, allowing a cache hit.
• This requires embedding models or other techniques to map diverse natural language inputs to a consistent key space, dramatically improving hit rates for conversational agents.

A cache that stores results for pure functions. A valid cache key must be derived from all function inputs, guaranteeing that the same key always maps to the same output.

• This is essential for caching computational results (e.g., mathematical transformations, data processing pipelines) where side effects are absent.
• It provides strong correctness guarantees: a cache hit is always valid. This contrasts with caches for external API data, where Time-To-Live (TTL) or explicit invalidation is needed due to external state changes.