Cost-Aware Prompt Caching Strategy Selector Prompt

This prompt is for platform engineers and AI infrastructure operators who need to make data-driven decisions about which parts of their prompts to cache. It analyzes prompt structure, request patterns, and cost models to produce a prioritized caching strategy with expected savings calculations. Use this when you have a stable set of system prompts or prefix templates, a known request volume, and you need to quantify the cost impact of caching before committing engineering time. This is not a prompt for runtime caching logic; it is a planning and analysis tool for infrastructure design.

The ideal user is someone who can provide concrete inputs: a representative prompt template with identifiable prefix and variable regions, estimated daily or monthly request volume, the per-token cost of their model (or a cost differential for cached vs. uncached tokens), and any known constraints on cache invalidation frequency. Without these inputs, the prompt will produce generic advice rather than an actionable cost model. The output is a structured recommendation that ranks prompt segments by caching priority, estimates token savings per request and in aggregate, and flags segments where caching is unlikely to break even due to high variability or low reuse.

Do not use this prompt when your prompt structure is still in flux, when you lack request volume data, or when you are evaluating a model provider that does not offer predictable caching behavior or pricing. It is also the wrong tool if you need a runtime decision—this prompt is for offline analysis. For runtime routing decisions based on cost or latency, use the Cost-Budget-Aware Router System Prompt Template or Latency-SLA Classification Prompt for Queue Assignment instead. After running this analysis, the next step is typically to implement the recommended caching strategy in your prompt assembly layer and validate actual savings against the projections.

This template is designed for platform engineers and infrastructure operators who need to decide which prompt prefixes to cache based on cost savings potential and cache hit probability. The prompt takes structured data about your prompt templates, request patterns, and cost parameters, then returns a prioritized list of caching recommendations with expected savings calculations. Use this when you have real request data and need to make caching decisions that balance storage costs against compute savings.

textCopy
You are a prompt caching strategy analyst for an AI infrastructure team. Your task is to analyze the provided prompt templates, request patterns, and cost data to produce a prioritized caching strategy.

## INPUT DATA

### Prompt Templates
[PROMPT_TEMPLATES]

### Request Pattern Data
[REQUEST_PATTERNS]

### Cost Parameters
- Cache storage cost per token per hour: [CACHE_STORAGE_COST]
- Compute cost per token (uncached): [COMPUTE_COST_UNCACHED]
- Compute cost per token (cached): [COMPUTE_COST_CACHED]
- Cache TTL in seconds: [CACHE_TTL]

### Constraints
- Maximum cache budget in tokens: [MAX_CACHE_BUDGET]
- Minimum expected savings threshold: [MIN_SAVINGS_THRESHOLD]

## OUTPUT SCHEMA

Return a JSON object with the following structure:
{
  "recommendations": [
    {
      "priority": number,
      "prompt_template_id": string,
      "prefix_to_cache": string,
      "prefix_token_count": number,
      "estimated_cache_hit_rate": number,
      "expected_daily_savings": number,
      "expected_monthly_savings": number,
      "cache_storage_cost_monthly": number,
      "net_monthly_savings": number,
      "break_even_hit_rate": number,
      "invalidation_frequency_estimate": string,
      "risk_assessment": string,
      "recommendation": string
    }
  ],
  "total_estimated_monthly_savings": number,
  "total_cache_storage_required_tokens": number,
  "cache_budget_utilization_percent": number,
  "unrecommended_prefixes": [
    {
      "prompt_template_id": string,
      "prefix": string,
      "reason_for_exclusion": string
    }
  ],
  "implementation_notes": [string]
}

## ANALYSIS INSTRUCTIONS

1. For each prompt template, identify the longest stable prefix that appears across multiple requests. A stable prefix is one that does not change between requests for the same template.

2. Calculate the expected cache hit rate based on the request pattern data. Consider:
   - Frequency of requests using this template
   - Variability in the non-prefix portion of the prompt
   - Time distribution of requests relative to cache TTL

3. Calculate expected savings:
   - Savings per cache hit = (prefix_token_count * (COMPUTE_COST_UNCACHED - COMPUTE_COST_CACHED))
   - Daily savings = savings_per_hit * estimated_daily_cache_hits
   - Monthly savings = daily_savings * 30
   - Cache storage cost = prefix_token_count * CACHE_STORAGE_COST * 24 * 30
   - Net monthly savings = monthly_savings - cache_storage_cost_monthly

4. Calculate break-even hit rate: the minimum hit rate at which caching this prefix becomes cost-effective.

5. Prioritize recommendations by net monthly savings in descending order.

6. Exclude prefixes where:
   - Net monthly savings is below MIN_SAVINGS_THRESHOLD
   - Cache hit rate is below break-even rate
   - Prefix is too short to justify caching overhead (less than [MIN_PREFIX_LENGTH] tokens)
   - Invalidation frequency would cause cache thrashing

7. Ensure total cache storage across all recommendations does not exceed MAX_CACHE_BUDGET. If it does, drop the lowest-priority recommendations until within budget.

8. For each recommendation, assess risks including:
   - Prefix drift if prompt templates change
   - Cache invalidation frequency and triggers
   - Cold start impact if cache is empty
   - Request pattern shifts over time

## OUTPUT REQUIREMENTS
- Return valid JSON only
- All monetary values in [CURRENCY]
- Token counts must be integers
- Rates and percentages as decimals (0.0 to 1.0)
- Include specific, actionable implementation notes

To adapt this template, replace each bracketed placeholder with your actual data. The [PROMPT_TEMPLATES] field should contain a structured description of each prompt template including its full text, identified stable prefixes, and template IDs. The [REQUEST_PATTERNS] field needs historical request data showing which templates are used, how frequently, and with what variability in the non-prefix portions. Set your cost parameters based on your model provider's pricing and your infrastructure costs. The [MIN_PREFIX_LENGTH] variable should be set based on your caching system's overhead characteristics—typically 20-50 tokens for most systems. Run this analysis periodically as request patterns shift and prompt templates evolve, not as a one-time exercise. The output JSON is designed to be consumed by automation that configures your caching layer, so maintain the schema contract strictly.

This prompt is designed to be called by an automated cost-optimization service, not a human chat interface. It should run as a scheduled or event-driven job—for example, triggered by a weekly analysis of prompt request logs or a CI/CD step before deploying a new prompt version. The harness must supply structured input about prompt structure, cache hit probability, storage costs, and invalidation frequency. The output is a machine-readable recommendation that a downstream service can act on to update cache configurations.

Input assembly: Collect the following data before calling the prompt: (1) a list of prompt prefixes with their token lengths and stability scores from your prompt management system; (2) cache hit probability estimates from your observability pipeline, calculated as the percentage of requests sharing each prefix over a representative time window; (3) cache storage cost per token from your inference provider's pricing page; (4) invalidation frequency estimates based on your prompt update cadence. Package these into the [PROMPT_STRUCTURE], [REQUEST_PATTERNS], [CACHE_STORAGE_COSTS], and [INVALIDATION_FREQUENCY] variables. Validation: Before calling the LLM, validate that token counts are positive integers, probabilities are between 0 and 1, and cost figures are non-negative floats. Reject any input that fails these checks and log the failure for debugging.

Output handling: Parse the model's JSON response and validate that each recommendation includes a prefix_id, cache_decision (boolean), expected_savings_tokens, and confidence score. If the JSON is malformed, retry once with a repair prompt that includes the raw output and the expected schema. Cost tracking: Log the token cost of running this selector prompt itself—it should be negligible compared to the savings it identifies, but track it to ensure the optimization loop doesn't become a cost center. Safe defaults: If the model returns no recommendations or the parsing fails after retries, default to caching nothing and alert the platform team. Never cache a prefix without a validated recommendation. Model choice: Use a fast, inexpensive model (e.g., GPT-4o-mini, Claude Haiku) for this task—the reasoning is straightforward arithmetic and comparison, not deep semantic analysis. Human review: Not required for routine runs, but flag recommendations where expected_savings_tokens exceeds a configurable threshold (e.g., 1M tokens/month) for a quick human sanity check before applying the cache configuration change.