Cache Stampede Root Cause Prompt

Platform engineers and SREs use this prompt to diagnose cache stampede or thundering herd incidents during post-incident review. The job-to-be-done is structured root cause analysis: turning a chaotic set of signals—cache hit ratio drops, key expiry spikes, recomputation latency surges, and lock contention—into a clear trigger event and a set of specific cache architecture remediation recommendations. The ideal user is an engineer who already has access to incident data from monitoring systems, cache observability tools, and application metrics, and needs to produce a defensible root cause document rather than a gut-feel guess.

The prompt expects structured incident data as input: cache hit ratios over the incident window, key TTL configurations, request concurrency metrics, recomputation latency distributions, and lock acquisition or contention logs. Without this data, the analysis will be speculative. Do not use this prompt for real-time mitigation decisions—it is designed for post-hoc root cause analysis, not live traffic control. It is also not a replacement for cache architecture design; it diagnoses what went wrong in an existing system rather than proposing a greenfield caching strategy. If you lack key access distribution data or lock contention metrics, the prompt will flag missing evidence rather than fabricating a cause.

Wire this prompt into an automated incident analysis pipeline where structured incident data is already aggregated, or use it manually during a post-incident review session. The output is a root cause hypothesis with supporting evidence, a confidence assessment, and specific remediation candidates such as probabilistic early expiration, external recompute orchestration, or lock timeout tuning. After receiving the output, validate the hypothesis against additional data sources before committing to remediation actions. For high-severity incidents affecting revenue or data integrity, always pair the prompt output with human SRE review before publishing the postmortem.

This template is designed to be copied directly into your AI harness, observability tool, or on-call runbook. It accepts structured incident data—cache metrics, key access patterns, recomputation costs, and lock contention signals—and returns a root cause hypothesis with specific remediation recommendations. The square-bracket placeholders represent the minimum required inputs; adapt them to match your telemetry sources and cache layer topology.

codeCopy
You are an SRE investigator specializing in distributed cache failures. Analyze the following incident data to determine the root cause of a cache stampede or thundering herd event.

[INCIDENT_TIMELINE]
[KEY_ACCESS_DISTRIBUTION]
[CACHE_EXPIRY_PATTERNS]
[RECOMPUTATION_COST_METRICS]
[LOCK_CONTENTION_SIGNALS]
[INFRASTRUCTURE_TOPOLOGY]

[OUTPUT_SCHEMA]
{
  "root_cause_hypothesis": "string",
  "trigger_event": "string",
  "contributing_factors": ["string"],
  "evidence": [
    {
      "source": "string",
      "observation": "string",
      "relevance": "string"
    }
  ],
  "stampede_mechanism": "synchronous_expiry | lock_contention | cold_start | hot_key_overload | cascading_failure",
  "affected_keys": ["string"],
  "remediation_recommendations": [
    {
      "action": "string",
      "category": "immediate | architectural | monitoring",
      "expected_impact": "string",
      "implementation_effort": "low | medium | high"
    }
  ],
  "confidence": "high | medium | low",
  "unexplained_observations": ["string"]
}

[CONSTRAINTS]
- Cite specific metrics, timestamps, and key names from the provided data.
- If lock contention is present, distinguish between client-side locking and server-side mutex contention.
- If recomputation cost data is provided, quantify the amplification factor (requests per recomputation).
- Do not speculate beyond the provided evidence; flag gaps in [unexplained_observations].
- If multiple mechanisms are possible, rank them by likelihood and explain your ranking.
- Recommend probabilistic early expiration (TTL jitter) only when synchronous expiry is confirmed.

To adapt this template, replace each placeholder with data from your observability stack. [INCIDENT_TIMELINE] should include deployment events, traffic spikes, and cache metric anomalies with timestamps. [KEY_ACCESS_DISTRIBUTION] requires per-key request rates and cardinality data—ingest this from your cache proxy or client-side metrics. [CACHE_EXPIRY_PATTERNS] needs TTL configurations and observed expiry events; pull these from cache server logs or configuration management. [RECOMPUTATION_COST_METRICS] should include p50/p99 latency and resource consumption for cache miss regeneration. [LOCK_CONTENTION_SIGNALS] requires mutex wait times, lock acquisition failures, or connection pool saturation metrics. [INFRASTRUCTURE_TOPOLOGY] should describe the cache layer architecture—client-side, reverse proxy, or distributed cluster—and any relevant replication or sharding configuration. For high-severity incidents, route the output through a human reviewer before applying remediation actions, especially when the recommendation involves cache configuration changes or lock implementation modifications.

This prompt is designed to be invoked during active cache degradation incidents or as part of a structured postmortem analysis. The primary integration point is your incident management bot or an internal SRE toolchain. The prompt expects a structured [INCIDENT_DATA] payload containing cache metrics, key access logs, and configuration snapshots. Before calling the model, your harness must assemble this payload from your observability stack—typically pulling time-series data from Prometheus or Datadog for cache hit ratios and recomputation latency, extracting key-level access patterns from Redis MONITOR logs or Elasticsearch, and capturing TTL configurations from your deployment manifests or config stores.

The implementation should enforce a strict validation layer on the input side. If the [INCIDENT_DATA] payload is missing any of the three required sections—cache_metrics, key_access_samples, or config_snapshot—the harness must abort and return a structured error asking the user to provide the missing data. On the output side, parse the model's response against the [OUTPUT_SCHEMA] defined in the prompt. Validate that the root_cause object contains a non-empty trigger_event and at least one evidence item. If the model returns a malformed JSON response, implement a single retry with a repair prompt that includes the raw output and the schema. Do not retry more than once for the same input; instead, log the failure and escalate to the on-call engineer for manual review. For high-severity incidents where a wrong diagnosis could delay mitigation, always route the final output to a human for approval before posting it to the incident timeline.

Choose a model with strong reasoning and structured output capabilities for this task. GPT-4o or Claude 3.5 Sonnet are appropriate defaults. Avoid smaller or faster models that may hallucinate causal links when the evidence is sparse. Wire the prompt into your workflow with a clear logging sidecar: capture the full prompt, the assembled [INCIDENT_DATA], the raw model response, and the parsed output. This audit trail is critical for postmortem reviews and for improving the prompt over time. Finally, ensure your harness handles timeouts gracefully—if the model call exceeds 30 seconds, abort and fall back to a pre-cached checklist of common stampede causes for the on-call engineer to review manually.