Pod Disruption Budget (PDB)

A typical PDB manifest for an agent deployment with 4 replicas, ensuring at least 3 are always available during voluntary disruptions:

yamlCopy
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: agent-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: inference-agent

An equivalent configuration using maxUnavailable:

yamlCopy
apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: inference-agent

Once applied, the PDB object has a status field that provides real-time information:

• currentHealthy: The number of pods currently observed as healthy and ready.
• desiredHealthy: The minimum number of pods required to be healthy, calculated from minAvailable or maxUnavailable.
• disruptionsAllowed: The most critical field. It shows how many pods can currently be disrupted without violating the budget. This value is 0 when the system is at its disruption limit.
• expectedPods: The total number of pods matched by the selector.

Platform engineers monitor disruptionsAllowed to understand the system's capacity for safe maintenance.

PDBs only govern voluntary disruptions. It is critical to understand the distinction:

• Voluntary Disruptions: Actions initiated by a cluster administrator or automated process that are expected and controlled. Examples include:

  • Draining a node for maintenance (kubectl drain).
  • A deployment update triggering a rolling update.
  • Manually deleting a pod. The orchestrator will respect the PDB, blocking the action if it would violate the budget.

• Involuntary Disruptions: Unplanned failures. Examples include:

  • A node hardware failure.
  • A pod eviction due to the node running out of resources.
  • A kernel panic. PDBs do not protect against these. Resilience here is provided by replica counts, self-healing, and anti-affinity rules.

When configuring PDBs for agent orchestration:

• Set Realistic Budgets: For a deployment with replicas: 4, maxUnavailable: 1 (25%) is common. For critical agents, use maxUnavailable: 0 or a high minAvailable percentage.
• Align with Replica Count: Ensure your PDB allows the orchestrator to make progress. A PDB with minAvailable: 100% on a deployment prevents all voluntary disruptions, which can block necessary updates.
• Use with Anti-Affinity: Combine PDBs with pod anti-affinity rules to ensure agent pods are spread across nodes. This prevents a single node drain from taking down multiple pods and violating the PDB.
• Monitor Budget Violations: Use orchestration observability tools to alert when disruptionsAllowed is 0 for extended periods, indicating a potential operational blocker.

A deployment strategy that incrementally replaces instances of an old agent version with a new version. This is a primary use case for a Pod Disruption Budget.

• The orchestrator (e.g., Kubernetes) terminates old pods and creates new ones in a controlled sequence.
• The PDB acts as a guardrail, ensuring the number of unavailable pods during this process never exceeds the defined threshold (e.g., maxUnavailable: 1). This ensures zero-downtime updates while maintaining service-level agreements.

The controlled shutdown process for an agent, allowing it to complete in-flight tasks and release resources. When a voluntary disruption (like a node drain) occurs, the orchestrator sends a SIGTERM signal to initiate this process.

• A Pod Disruption Budget influences the timing of these terminations by limiting how many can happen concurrently.
• The agent has a terminationGracePeriodSeconds to finish its work before receiving a SIGKILL. This process is critical for preventing data corruption and ensuring clean handoffs.

Administrative operations that prepare a node for maintenance, directly interacting with Pod Disruption Budgets.

• Cordon: Marks a node as unschedulable, preventing new pods from being placed on it.
• Drain: Safely evicts all pods from a node, respecting each pod's PDB. The drain command will block if evicting a pod would violate its PDB. These commands are used during node updates, scaling down, or hardware repairs, making PDBs essential for planned cluster operations.

A classification (Guaranteed, Burstable, BestEffort) assigned by an orchestrator based on resource requests and limits. While PDBs protect against voluntary disruptions, QoS influences behavior during involuntary resource pressure:

• Under memory pressure, the system may evict pods to reclaim resources.
• BestEffort pods (with no requests/limits) are evicted first, followed by Burstable, then Guaranteed. Understanding QoS is crucial for designing systems where PDBs alone are insufficient for overall resilience.