Resource Quota Exceeded Recovery Prompt

This prompt is for cluster operators and platform engineers who must resolve a ResourceQuota or LimitRange rejection in Kubernetes without manually recalculating every container's request. The job-to-be-done is straightforward: consume the rejected pod specification and the exact API server error message, then produce a corrected pod spec or a specific quota adjustment that will pass admission control. The ideal user is someone who understands Kubernetes scheduling concepts but wants to offload the arithmetic of fitting a workload into constrained namespace budgets to a reliable, repeatable prompt.

Use this prompt when the failure is a deterministic resource accounting problem—CPU, memory, ephemeral storage, or object count limits—and you have the full error text and the original manifest. Do not use it for generic scheduling failures (e.g., node affinity mismatches, taint tolerations, or persistent volume binding errors) because those require different diagnostic context. The prompt assumes the quota or limit range is the sole rejection cause; if multiple admission plugins are failing, resolve them one at a time to keep the correction scope narrow and auditable.

The prompt is designed for a harness that can validate the output against the target namespace's actual ResourceQuota and LimitRange objects before applying it. It is not a substitute for capacity planning or cost governance—it only solves the immediate admission failure. Before wiring this into an automated retry loop, ensure your harness enforces a retry budget and escalates to a human if the suggested adjustment would exceed a hard cluster-wide resource ceiling or if the prompt produces a correction that still fails validation after two attempts.

The prompt below is designed to be copied directly into your AI harness. It expects a rejected Kubernetes pod specification and the exact error message from the API server. The model is instructed to reason about the failure, propose a minimal adjustment to either the pod's resource requests and limits or the namespace-level ResourceQuota or LimitRange, and output the correction in a structured format that your controller can parse and apply. All placeholders use square brackets for safe template substitution.

textCopy
You are a Kubernetes resource recovery assistant. Your task is to analyze a pod admission failure caused by resource quota or limit range constraints and propose a minimal, valid correction.

## INPUT
- Rejected Pod Specification (YAML):
[REJECTED_POD_SPEC]

- API Server Error Message:
[API_ERROR_MESSAGE]

- Namespace ResourceQuota (YAML, if available):
[RESOURCE_QUOTA_YAML]

- Namespace LimitRange (YAML, if available):
[LIMIT_RANGE_YAML]

## CONSTRAINTS
- Propose the smallest change that satisfies the constraint.
- Prefer adjusting the pod's resource requests and limits over modifying the quota, unless the quota is clearly misconfigured.
- Do not remove required containers or volumes.
- If the error is a LimitRange violation, ensure the corrected pod spec complies with min, max, and default values.
- If the error is a ResourceQuota violation, ensure the corrected pod spec stays within the remaining quota headroom.
- If the quota itself is too low for any valid pod, explain why and propose a quota increase with justification.
- Never suggest disabling admission controllers or removing quota enforcement.

## OUTPUT_SCHEMA
Return a JSON object with the following structure:
{
  "analysis": "Brief explanation of the failure cause.",
  "correction_type": "pod_spec" | "quota" | "limit_range",
  "corrected_yaml": "The full corrected YAML document.",
  "diff_summary": "Human-readable summary of what changed and why.",
  "validation_notes": "Any assumptions made or risks to verify before applying."
}

## EXAMPLES

Example 1: LimitRange CPU violation
Input Error: "pod rejected: max cpu limit per container is 2"
Rejected Pod Spec: container requests cpu 3
Output:
{
  "analysis": "Container CPU request of 3 exceeds LimitRange max of 2.",
  "correction_type": "pod_spec",
  "corrected_yaml": "... pod spec with cpu request reduced to 2 ...",
  "diff_summary": "Reduced container CPU request from 3 to 2 to comply with LimitRange.",
  "validation_notes": "Verify that 2 CPU is sufficient for the workload under expected load."
}

Example 2: ResourceQuota memory exhausted
Input Error: "pod rejected: exceeded quota: requests.memory"
Rejected Pod Spec: container requests memory 8Gi
ResourceQuota: hard limits.memory: 16Gi, current usage 12Gi
Output:
{
  "analysis": "Namespace memory quota has 4Gi remaining. Pod requests 8Gi.",
  "correction_type": "pod_spec",
  "corrected_yaml": "... pod spec with memory request reduced to 4Gi ...",
  "diff_summary": "Reduced memory request from 8Gi to 4Gi to fit within remaining quota headroom.",
  "validation_notes": "Confirm the workload can operate with 4Gi. If not, a quota increase to at least 20Gi is required."
}

## RISK_LEVEL
[RISK_LEVEL]

If RISK_LEVEL is "high" (production namespace), include a warning that human review is required before applying the correction and that workload performance must be validated in a staging environment first.

To adapt this template, replace each square-bracket placeholder with data from your admission webhook or controller. The [REJECTED_POD_SPEC] should be the full YAML of the pod that failed admission. The [API_ERROR_MESSAGE] should be the exact error string from the Kubernetes API server, typically found in the event stream or admission review response. The [RESOURCE_QUOTA_YAML] and [LIMIT_RANGE_YAML] placeholders are optional but strongly recommended—providing them prevents the model from guessing at namespace constraints and reduces the chance of proposing an invalid correction. The [RISK_LEVEL] placeholder should be set to "high" for production namespaces and "low" for development or staging environments, which controls whether the output includes a mandatory human-review warning.

Before wiring this prompt into an automated retry loop, test it against a curated set of failure scenarios: a LimitRange CPU violation, a LimitRange memory-to-request ratio violation, a ResourceQuota exhaustion where the pod can be downsized, and a ResourceQuota exhaustion where no valid pod fits. For each scenario, validate that the corrected_yaml field parses as valid YAML, that the correction_type matches the actual fix, and that the corrected pod spec would pass admission against the provided quota and limit range. If the model proposes a quota increase when a pod downsizing was possible, flag that as a failure—the prompt's constraint ordering prefers pod adjustment over quota modification. Run these evaluations before every prompt version change and log the results for audit.

This prompt is designed to sit inside a Kubernetes operator, admission webhook, or CI/CD pipeline stage that catches ResourceQuota or LimitRange rejection events. The harness should intercept the rejected pod spec and the API server error message, inject them into the prompt's [REJECTED_SPEC] and [QUOTA_ERROR] placeholders, and optionally supply the namespace's current [QUOTA_DEFINITION] and [LIMIT_RANGE_DEFINITION] for full constraint awareness. Do not call this prompt for every scheduling failure—only for events where the error message explicitly references resource quota exceeded, limit range constraints, or request/limit violations. Routing the wrong failure type into this prompt will produce irrelevant suggestions and waste inference tokens.

The harness must validate the model's output before applying any suggested changes. Parse the response for a valid Kubernetes resource manifest in the corrected_spec field and confirm it passes kubectl --dry-run=server or an equivalent schema validation against the target cluster's API version. If the model suggests increasing a quota rather than reducing pod requests, flag the output for human review unless the operator's service account has quota-modification permissions. Implement a retry budget of two attempts: if the first corrected spec still fails admission, feed the new error back into the prompt with the previous suggestion as additional context, then escalate to a human operator if the second attempt also fails. Log every attempt, the original error, the model's suggestion, and the validation result for auditability.

For production deployments, prefer a model with strong structured-output support (GPT-4o, Claude 3.5 Sonnet, or a fine-tuned open-weight model) and enforce JSON mode or function-calling output to guarantee parseable corrected_spec fields. If the cluster runs in an air-gapped environment, use a local model with the same structured-output constraint and test against a representative quota configuration before deployment. Never automatically apply quota increases without human approval—this prompt's safest path is reducing pod resource requests to fit within existing bounds. Wire the harness to emit a Prometheus metric counting recovery attempts, successes, and escalations so the team can track whether quota exhaustion is a transient scheduling issue or a systemic capacity signal that requires cluster expansion.