Node Scheduling Failure Recovery Prompt

This prompt is designed for platform engineers and SRE teams who need to diagnose and resolve pod scheduling failures in Kubernetes clusters. The core job-to-be-done is to take a pending pod's specification and the cluster's current node conditions, then propose a corrected set of scheduling rules—such as node affinity, tolerations, or topology spread constraints—that will allow the pod to be placed. It is not a general-purpose Kubernetes assistant; it is specifically scoped to failures caused by affinity/anti-affinity mismatches, taint/toleration conflicts, and topology constraint violations.

Use this prompt when you have a concrete scheduling failure event: a pod stuck in Pending status with a clear FailedScheduling event message, and you can provide both the pod manifest and a snapshot of relevant node labels and taints. The prompt requires structured input, including the pod's current scheduling directives and the target node pool's topology. Do not use this prompt for generic pod startup failures, image pull errors, crash loops, or resource exhaustion issues where the scheduler is not the root cause. It is also unsuitable for clusters with custom schedulers that use non-standard plugin logic.

Before invoking this prompt, ensure you have extracted the exact scheduling error from kubectl describe pod and have a sanitized view of node metadata. The output should be treated as a suggested configuration, not an auto-applied fix. Always validate the proposed scheduling rules against your cluster's admission controllers and policy engines (e.g., OPA/Gatekeeper) before applying. For production clusters, route the generated diff through a peer review or a GitOps pull request rather than applying it directly.

This prompt template is designed to be wired into an automated recovery harness. It receives a pending pod's specification and the current state of candidate nodes, then produces a corrected scheduling configuration. The output is structured JSON, making it suitable for direct consumption by a control loop or a human operator reviewing a suggested fix. The template uses square-bracket placeholders for all dynamic inputs, ensuring it can be adapted to different clusters and failure contexts without modification.

textCopy
You are an expert Kubernetes scheduler debugger. Your task is to analyze a pod scheduling failure and propose a corrected pod specification or node configuration that will allow the pod to be scheduled.

## INPUT
- Pending Pod Specification (YAML):
[PENDING_POD_YAML]

- Candidate Node Conditions (JSON list of node names, taints, labels, and allocatable resources):
[NODE_CONDITIONS_JSON]

- Scheduler Error Message (plain text from `kubectl describe pod`):
[SCHEDULER_ERROR_MESSAGE]

## CONSTRAINTS
- Do not change the pod's container images, command, or core application logic.
- Prefer the least invasive fix: adjust pod scheduling directives before suggesting node-level changes.
- If a node-level change is the only option, explicitly flag it as a cluster-admin action.
- Ensure the proposed fix does not violate any explicit anti-affinity rules already in the pod spec.
- If the failure is due to a resource shortage (CPU, memory), suggest a specific, minimal reduction in requests that aligns with the node's allocatable capacity.

## OUTPUT_SCHEMA
Respond with a single JSON object conforming to this structure:
{
  "diagnosis": "A concise, one-sentence summary of the root cause.",
  "proposed_fix_type": "One of: 'pod_spec_patch', 'node_label_patch', 'node_taint_patch', 'resource_adjustment', 'human_escalation'.",
  "corrected_pod_spec_yaml": "The full, corrected pod YAML if fix_type is 'pod_spec_patch' or 'resource_adjustment'; otherwise, null.",
  "node_patch_commands": ["A list of kubectl patch or taint commands if fix_type is 'node_label_patch' or 'node_taint_patch'; otherwise, an empty list."],
  "explanation": "A step-by-step explanation of why the fix resolves the specific error, referencing the constraints that were violated.",
  "risk_assessment": "One of: 'low', 'medium', 'high'. 'high' if the fix could affect other workloads on the node."
}

## EXAMPLES
[FEW_SHOT_EXAMPLES]

## RISK_LEVEL
[RISK_LEVEL]

To adapt this template, replace the placeholders with data from your cluster's API. The [PENDING_POD_YAML] should be the exact output of kubectl get pod <name> -o yaml. The [NODE_CONDITIONS_JSON] is a critical input; construct it by querying nodes with kubectl get nodes -o json and filtering for the fields relevant to scheduling: spec.taints, metadata.labels, and status.allocatable. The [FEW_SHOT_EXAMPLES] placeholder should be populated with 2-3 worked examples of common scheduling failures and their correct fixes to improve model accuracy. For high-risk environments, set [RISK_LEVEL] to high to instruct the model to prefer the human_escalation fix type when node taints must be altered. The next step is to integrate this prompt into a validation harness that checks the proposed YAML against the target cluster's API version schema before applying any changes.

This prompt is designed to be called within an automated triage pipeline, not as a standalone chat interaction. The harness should trigger when a pod remains in a Pending state beyond a configurable threshold (e.g., 5 minutes) and the scheduling event reason matches known failure classes (Unschedulable, FailedScheduling). The input assembly step must collect the full pod spec (kubectl get pod <name> -o yaml), the node list with taints and conditions (kubectl get nodes -o json), and the specific scheduling error message from kubectl describe pod. This context is packed into the [POD_SPEC], [NODE_CONDITIONS], and [SCHEDULING_ERROR] placeholders before the prompt is rendered.

After the model returns a proposed correction, the harness must not apply it automatically. First, validate the output structure: confirm the corrected_scheduling_rules block contains valid nodeSelectorTerms, affinity, or tolerations fields. Second, run a satisfiability check by using a dry-run scheduler or a lightweight constraint solver that tests whether at least one existing node matches the proposed rules. If no node satisfies the corrected constraints, increment a retry counter and re-invoke the prompt with the unsatisfiability reason appended to the error context. Implement a hard retry budget of 3 attempts; after the budget is exhausted, escalate to a human operator via your incident management system with the full chain of attempts and the original pod state.

Model choice matters here. Use a model with strong structured-output capabilities and a context window large enough to hold multiple node definitions. For production, prefer a model that supports JSON mode or function calling so you can enforce the output schema at the API level rather than relying solely on prompt instructions. Log every invocation—input context, raw output, validation result, and retry count—to your observability platform. This trace is essential for debugging prompt drift, model refusals, or unexpected constraint suggestions. Never apply the corrected scheduling rules directly to a running cluster without human approval unless the pod is in a pre-production namespace and the change is limited to soft preferences (preferredDuringScheduling) rather than hard requirements.