Cache Admission Policy

A cache admission policy is fundamentally a gatekeeper, deciding whether to admit a new item. This is distinct from an eviction policy, which decides which existing item to remove when the cache is full. They work in tandem: admission controls inflow, eviction controls outflow. For optimal performance, a system might use a Frequency-Based Admission policy (like 2Q or TinyLFU) to only admit items predicted to be accessed again, while using LRU for eviction.

These policies admit items based on predicted future access frequency, not just recency. A common implementation is the TinyLFU sketch, which uses a compact probabilistic structure to track access counts.

• Mechanism: A new item is only admitted if its estimated frequency is higher than the item the eviction policy would remove (e.g., the LRU victim).
• Benefit: Prevents one-hit wonders—items accessed only once—from polluting the cache and evicting potentially valuable data.
• Use Case: Highly effective for workloads with a mix of popular and tail items, common in web object and database query caching.

This characteristic considers the size of the candidate item relative to the benefit of caching it. The core principle is that caching a very large item has a high opportunity cost, as it displaces many smaller, potentially more frequently accessed items.

• Simple Rule: Do not admit items larger than a certain percentage of the total cache capacity.
• Advanced Rule: Use a cost-benefit model, such as size / access frequency. A large, rarely accessed file would be rejected.
• Impact: Crucial for caches storing variable-sized objects (e.g., images, API responses, model outputs) to maximize the aggregate hit ratio and utility per byte.

A strict policy that only admits the results of pure, deterministic functions. A cache hit is guaranteed to be valid if the function inputs are identical.

• Key Property: The function must have no side effects and its output must depend solely on its input arguments.
• Implementation: The cache key is a hash of the function signature and its serialized arguments.
• Use Case: Ideal for agent-side caching of computational results, such as:
  • Mathematical transformations.
  • Semantic cache lookups where the same query intent yields the same LLM inference.
  • Processing steps within a deterministic agent workflow.

This policy incorporates the cost of a cache miss into the admission decision. The goal is to prioritize caching items that are expensive to recompute or fetch.

• Cost Factors: Can include latency, financial API call costs, computational load, or load on a backend system.
• Logic: An item with a high retrieval cost and moderate access frequency may be admitted over a low-cost, high-frequency item.
• Example: In an AI agent, the result of a slow, paid external API call is a prime candidate for admission, even if it will only be reused a few times, because the cost of a miss is high.

In a hierarchical cache (e.g., L1/L2), admission policies can differ per tier to optimize the overall memory hierarchy.

• L1 Cache (Small/Fast): May have a very selective policy (e.g., Frequency-Based), admitting only the "hottest" items to keep its limited space maximally effective.
• L2 Cache (Large/Slower): May use a more permissive admission policy, acting as a victim cache for L1 evictions or admitting items with longer-term value.
• Orchestration: A miss in L1 that is fetched from the primary source might be admitted to L2 but not necessarily back into L1, depending on the tier-specific policy. This structure is common in CPU architectures and can be applied to distributed cache systems.

A cache eviction policy is the algorithm that determines which item to remove from a full cache to make space for a new entry. It works in tandem with the admission policy to manage cache capacity. Common algorithms include:

• LRU (Least Recently Used): Evicts the item accessed longest ago.
• LFU (Least Frequently Used): Evicts the item with the fewest accesses.
• FIFO (First-In, First-Out): Evicts the oldest item by insertion time. The choice of eviction policy directly impacts the cache's hit ratio and overall performance.

The cache hit ratio is the primary metric for cache performance, calculated as (Number of Cache Hits / Total Requests) * 100%. A high ratio indicates the cache is effectively storing useful data, reducing calls to slower primary sources like APIs or databases. This metric is influenced by both the admission policy (what gets in) and the eviction policy (what stays in). Performance engineers optimize these policies to maximize this ratio for a given cache size.

A semantic cache stores computed results (like LLM responses or API call outputs) based on the meaning or intent of a query, rather than an exact string match of the request. This allows for cache hits on semantically similar but not identical requests. For example, queries for "capital of France" and "what is the main city in France?" could yield a cache hit. Admission policies for semantic caches are complex, as they must evaluate the semantic similarity of a new query to existing cached entries.

A deterministic cache stores the results of pure, side-effect-free functions or API calls where the same input arguments always produce an identical output. This guarantees that a cached result remains valid for future identical requests. Admission for a deterministic cache is straightforward: if the computation is pure and the inputs match, the result is cacheable. This is foundational for caching within AI agent tool-calling, where repeated calls with the same parameters should yield the same result.

Cache invalidation is the process of marking cached data as stale or removing it entirely to ensure subsequent requests fetch fresh data from the primary source. It is critical for maintaining cache consistency. Common strategies include:

• Time-To-Live (TTL): Automatic expiration after a set duration.
• Explicit Invalidation: Manually purging entries when source data changes.
• Write-Through/Write-Behind: Updating the cache synchronously or asynchronously on writes. Admission policies must consider invalidation costs; caching highly volatile data may be counterproductive.

The cache-aside pattern (or lazy loading) is a common caching strategy where the application code (e.g., the AI agent) explicitly manages the cache. The flow is:

1. On a request, the agent first checks the cache.
2. If a cache hit occurs, it uses the cached data.
3. If a cache miss occurs, it fetches from the primary source (API/database), then applies the admission policy to decide whether to insert the result into the cache. This pattern gives the agent direct control over admission logic based on business rules or cost.