Pod Disruption Budget (PDB)

A PDB is defined by one of two mutually exclusive parameters in its spec:

• minAvailable: Specifies the absolute number or percentage of pods from the controlled set that must remain available during a disruption. For example, minAvailable: 2 or minAvailable: "50%".
• maxUnavailable: Specifies the absolute number or percentage of pods from the controlled set that can be unavailable during a disruption. For example, maxUnavailable: 1 or maxUnavailable: "25%".

These parameters are evaluated against the total number of pods matched by the selector. You must define only one of these fields.

The selector field is a label selector that determines which pods the PDB governs. It uses the same syntax as other Kubernetes selectors (e.g., matchLabels, matchExpressions).

• The PDB only protects pods whose labels match this selector.
• It is crucial that the selector accurately targets the pods belonging to your application's deployment, statefulset, or other controller.
• A common pattern is to use the same selector as the parent workload. For example, a PDB for a deployment with app: my-agent would use selector: { matchLabels: { app: my-agent } }.

A typical PDB manifest for an agent deployment with 4 replicas, ensuring at least 3 are always available during voluntary disruptions:

yamlCopy
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: agent-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: inference-agent

An equivalent configuration using maxUnavailable:

yamlCopy
apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: inference-agent

Once applied, the PDB object has a status field that provides real-time information:

• currentHealthy: The number of pods currently observed as healthy and ready.
• desiredHealthy: The minimum number of pods required to be healthy, calculated from minAvailable or maxUnavailable.
• disruptionsAllowed: The most critical field. It shows how many pods can currently be disrupted without violating the budget. This value is 0 when the system is at its disruption limit.
• expectedPods: The total number of pods matched by the selector.

Platform engineers monitor disruptionsAllowed to understand the system's capacity for safe maintenance.

PDBs only govern voluntary disruptions. It is critical to understand the distinction:

• Voluntary Disruptions: Actions initiated by a cluster administrator or automated process that are expected and controlled. Examples include:

  • Draining a node for maintenance (kubectl drain).
  • A deployment update triggering a rolling update.
  • Manually deleting a pod. The orchestrator will respect the PDB, blocking the action if it would violate the budget.

• Involuntary Disruptions: Unplanned failures. Examples include:

  • A node hardware failure.
  • A pod eviction due to the node running out of resources.
  • A kernel panic. PDBs do not protect against these. Resilience here is provided by replica counts, self-healing, and anti-affinity rules.

When configuring PDBs for agent orchestration:

• Set Realistic Budgets: For a deployment with replicas: 4, maxUnavailable: 1 (25%) is common. For critical agents, use maxUnavailable: 0 or a high minAvailable percentage.
• Align with Replica Count: Ensure your PDB allows the orchestrator to make progress. A PDB with minAvailable: 100% on a deployment prevents all voluntary disruptions, which can block necessary updates.
• Use with Anti-Affinity: Combine PDBs with pod anti-affinity rules to ensure agent pods are spread across nodes. This prevents a single node drain from taking down multiple pods and violating the PDB.
• Monitor Budget Violations: Use orchestration observability tools to alert when disruptionsAllowed is 0 for extended periods, indicating a potential operational blocker.

A periodic diagnostic probe used by an orchestration system to determine if an agent is functioning correctly. A failing health check can trigger a self-healing action, such as a pod restart. This is distinct from a PDB, which governs voluntary disruptions. Key types include:

• Liveness Probe: Determines if the agent container is running. Failure results in a restart.
• Readiness Probe: Determines if the agent is ready to accept traffic. Failure removes the pod from service load balancers.

An orchestration capability where the system automatically detects and recovers from agent failures. This typically works in tandem with health checks and PDBs:

• Self-healing handles involuntary disruptions (crashes, node failures) by restarting or rescheduling pods.
• A Pod Disruption Budget (PDB) protects against voluntary disruptions (maintenance, updates) by limiting how many pods can be down at once. Together, they ensure high availability across both planned and unplanned downtime scenarios.

A deployment strategy that incrementally replaces instances of an old agent version with a new version. This is a primary use case for a Pod Disruption Budget.

• The orchestrator (e.g., Kubernetes) terminates old pods and creates new ones in a controlled sequence.
• The PDB acts as a guardrail, ensuring the number of unavailable pods during this process never exceeds the defined threshold (e.g., maxUnavailable: 1). This ensures zero-downtime updates while maintaining service-level agreements.

The controlled shutdown process for an agent, allowing it to complete in-flight tasks and release resources. When a voluntary disruption (like a node drain) occurs, the orchestrator sends a SIGTERM signal to initiate this process.

• A Pod Disruption Budget influences the timing of these terminations by limiting how many can happen concurrently.
• The agent has a terminationGracePeriodSeconds to finish its work before receiving a SIGKILL. This process is critical for preventing data corruption and ensuring clean handoffs.

Administrative operations that prepare a node for maintenance, directly interacting with Pod Disruption Budgets.

• Cordon: Marks a node as unschedulable, preventing new pods from being placed on it.
• Drain: Safely evicts all pods from a node, respecting each pod's PDB. The drain command will block if evicting a pod would violate its PDB. These commands are used during node updates, scaling down, or hardware repairs, making PDBs essential for planned cluster operations.

A classification (Guaranteed, Burstable, BestEffort) assigned by an orchestrator based on resource requests and limits. While PDBs protect against voluntary disruptions, QoS influences behavior during involuntary resource pressure:

• Under memory pressure, the system may evict pods to reclaim resources.
• BestEffort pods (with no requests/limits) are evicted first, followed by Burstable, then Guaranteed. Understanding QoS is crucial for designing systems where PDBs alone are insufficient for overall resilience.