Kubernetes Pod Eviction Root Cause Analysis Prompt

This prompt is designed for platform engineers and SREs who are actively investigating an unexpected pod termination and have already exhausted the immediate surface-level details from kubectl describe pod. The job-to-be-done is to move from observing a terminated pod to understanding the causal chain of cluster-level decisions that led to the eviction. The ideal user has collected raw Kubernetes event streams, node condition statuses, and resource metrics, and now needs a structured differential diagnosis to decide whether the fix involves adjusting resource limits, modifying scheduling constraints, or escalating to a cluster administrator. The prompt instructs the LLM to act as a Kubernetes incident analyst, correlating pod status fields, event streams, node conditions, and cluster configurations to classify the eviction reason into categories like resource pressure, node condition changes, taint/toleration mismatches, priority preemption, or API-initiated evictions.

Before using this prompt, ensure you have gathered the complete output of kubectl get events for the namespace and timeframe of the incident, the full YAML output of the terminated pod (kubectl get pod <name> -o yaml), and the node description (kubectl describe node <node-name>) for the node where the pod was last scheduled. The prompt is most effective when the eviction reason is not immediately obvious—for example, when the pod status shows Evicted but the events are ambiguous, or when multiple pods are being terminated in a pattern that suggests a cluster-wide condition. It is not a replacement for real-time monitoring dashboards or alerting systems; it is a forensic analysis tool for post-incident investigation.

Do not use this prompt when the pod failure is clearly an application-level crash (e.g., a non-zero exit code from the container process) without any Kubernetes eviction signal. In those cases, a stack trace or log analysis prompt is more appropriate. Similarly, avoid this prompt if you have not yet collected the raw Kubernetes state—feeding incomplete or guessed data will produce a plausible but incorrect eviction chain. The output should be treated as a high-confidence hypothesis that still requires validation against your cluster's actual configuration and metrics before any remediation is applied. For high-severity production incidents, always pair the LLM's analysis with a human review of the proposed prevention strategy before making changes to resource limits, affinity rules, or priority classes.

This template is designed to be pasted directly into your AI harness. It expects structured input data collected from your cluster before the investigation begins. Replace every square-bracket placeholder with real values gathered from kubectl describe pod, kubectl get events, node conditions, and the cluster's resource metrics. The prompt forces the model to correlate pod-level signals with node-level and cluster-level state before producing a conclusion.

textCopy
You are an SRE assistant specializing in Kubernetes pod lifecycle analysis.

Analyze the following pod eviction event and determine the root cause.

## POD CONTEXT
- Pod Name: [POD_NAME]
- Namespace: [NAMESPACE]
- Node Name: [NODE_NAME]
- Priority Class: [PRIORITY_CLASS]
- QoS Class: [QOS_CLASS]
- Container Resource Requests: [CONTAINER_REQUESTS]
- Container Resource Limits: [CONTAINER_LIMITS]

## EVICTION EVENT
- Eviction Reason (from pod status): [EVICTION_REASON]
- Eviction Timestamp: [EVICTION_TIMESTAMP]
- Termination Message: [TERMINATION_MESSAGE]

## NODE STATE AT EVICTION TIME
- Node Conditions (Ready, MemoryPressure, DiskPressure, PIDPressure): [NODE_CONDITIONS]
- Allocatable Resources: [NODE_ALLOCATABLE]
- Capacity: [NODE_CAPACITY]
- Recent Node Events (last 15 minutes): [NODE_EVENTS]

## CLUSTER-WIDE CONTEXT
- Other Pods Evicted on Same Node (last 10 minutes): [CO_EVICTED_PODS]
- Cluster Autoscaler Events (if any): [AUTOSCALER_EVENTS]
- Recent Taints Applied to Node: [RECENT_TAINTS]

## OUTPUT REQUIREMENTS
Produce a JSON object with the following schema:
{
  "eviction_classification": "ResourcePressure | NodeCondition | TaintToleration | PriorityPreemption | Unknown",
  "primary_cause": "One-sentence root cause explanation grounded in the provided data.",
  "evidence_chain": [
    "Step-by-step reasoning linking pod state, node state, and cluster events to the conclusion."
  ],
  "prevention_strategy": {
    "immediate_action": "What to do now to prevent recurrence on this node.",
    "long_term_fix": "Configuration, resource, or scheduling policy change needed.",
    "monitoring_improvement": "Alert or dashboard that would have caught this earlier."
  },
  "confidence": "High | Medium | Low",
  "missing_data": ["List any data points that would increase confidence if available."]
}

## CONSTRAINTS
- Do not invent events, metrics, or node conditions not present in the input.
- If the eviction reason is ambiguous, state the leading hypothesis and what would rule it out.
- If priority preemption is suspected, explicitly name the preempting pod and its priority.
- If the data is insufficient for a High-confidence classification, set confidence accordingly and list what's missing.

Adaptation notes: If your cluster runs a custom scheduler or uses descheduler policies, add a [SCHEDULER_POLICY] placeholder and include relevant descheduler logs. For air-gapped environments where the model cannot access live cluster APIs, this prompt must be fed entirely from pre-collected data. The output schema is designed to be machine-readable for downstream automation—if a human is the primary consumer, consider adding a narrative_summary field. Always validate the JSON output against the schema before acting on the immediate_action field, especially in production clusters where a bad recommendation could trigger unnecessary node drains.

This prompt is designed to be called programmatically during an active incident or as part of an automated post-eviction analysis pipeline. The primary integration point is a webhook receiver that triggers on Evicted or Failed pod events from the Kubernetes API, or from monitoring systems like Prometheus AlertManager when pod disruption metrics spike. The harness must gather the required inputs—pod manifest, events, node conditions, and resource metrics—before invoking the model, because the prompt's diagnostic accuracy depends entirely on the completeness and freshness of this context.

Build a thin orchestration layer that queries the Kubernetes API for the evicted pod's last known spec, the node it was scheduled on, all events in the pod's namespace filtered by the pod name, and the node's conditions and allocatable capacity at the time of eviction. If your cluster runs metrics-server or Prometheus, also pull the pod's recent CPU/memory usage and the node's pressure metrics. Assemble these into the [EVIDENCE] placeholder as structured JSON or YAML. Validation step: before calling the model, verify that the evidence payload contains at least the pod phase, eviction reason, node name, and a timestamp. If any required field is missing, abort the model call and surface a data-gathering failure to the operator instead of generating an unreliable analysis.

For model selection, prefer a model with strong structured reasoning and JSON output capabilities, such as Claude 3.5 Sonnet or GPT-4o, because the prompt requires multi-hypothesis comparison and evidence weighting. Set temperature=0 or very low to maximize reproducibility. The output schema defined in the prompt should be enforced via the model's JSON mode or structured output API, not via post-hoc regex parsing. Retry logic: if the model returns a malformed JSON response or fails to populate the eviction_chain array, retry once with a simplified instruction appended: 'Your previous response was invalid. Return only valid JSON matching the output schema.' If the second attempt fails, escalate to a human responder with the raw evidence and the failed model output attached.

Log every invocation with the evidence payload, model response, and a correlation ID tied to the incident or alert. This audit trail is critical for postmortem reviews and for tuning the prompt over time. Human review gate: for any eviction classified as PriorityPreemption or NodePressure with a confidence score below 0.7, route the analysis to an on-call engineer for confirmation before auto-remediating. Auto-remediation—such as adjusting resource limits, adding node selectors, or cordoning a node—should only trigger when the eviction classification confidence exceeds 0.85 and the recommended action is idempotent. Never auto-apply taint changes or priority class modifications without human approval.

Finally, treat this prompt as a diagnostic tool, not a replacement for cluster-level observability. It works best when paired with a runbook that already defines standard responses to common eviction types. Use the model's output to accelerate the investigation, but always cross-reference its findings against your cluster's actual state before taking action. The most common production failure mode is stale evidence: if the harness queries the API too late and the node or pod state has changed, the model will produce a plausible but incorrect root cause. Mitigate this by timestamping all evidence and discarding any data older than the eviction event by more than 60 seconds.