Horizontal Pod Autoscaling (HPA)

HPA natively monitors CPU utilization and memory consumption per pod against defined target thresholds. When the average CPU usage across all pods exceeds 80% of the requested limit, the controller calculates the desired replica count using the formula: desiredReplicas = ceil(currentReplicas * (currentMetricValue / desiredMetricValue)). This ensures each pod maintains sufficient headroom to handle request spikes without throttling.

• Target utilization: Typically set to 70-80% for CPU, leaving buffer for sudden traffic bursts
• Stabilization window: Default 5-minute cooldown prevents rapid scale-down oscillations after brief load spikes
• Resource requests: Must be explicitly defined on pod templates; HPA cannot scale pods without known resource baselines

Beyond CPU and memory, HPA supports custom metrics exposed through the custom.metrics.k8s.io API, enabling scaling decisions based on application-specific signals like requests-per-second, queue depth, or inference latency. The Prometheus Adapter is commonly deployed to translate Prometheus queries into Kubernetes metrics, allowing HPA to scale model replicas based on P99 prediction latency or active gRPC stream count.

• Pod metrics: Per-pod values like http_requests_per_second averaged across all replicas
• Object metrics: Cluster-level signals like message_queue_length from an external queue
• External metrics: Non-Kubernetes signals such as cloud provider load balancer metrics or custom monitoring systems

While native HPA cannot scale below minReplicas (typically 1), integration with KEDA (Kubernetes Event-Driven Autoscaling) enables true scale-to-zero behavior. KEDA acts as a metrics server that can scale deployments to zero replicas when no inference requests are present and rapidly scale back up when traffic arrives. This is critical for GPU-accelerated model serving where idle instances incur significant cloud compute costs.

• Cold start mitigation: Combine with lightweight warm-up containers or model pre-loading scripts
• Activation lag: Expect 5-15 seconds for pod initialization plus model weight loading time
• Cost optimization: Reduces GPU compute costs by 60-80% for bursty inference workloads with long idle periods

The HPA controller uses a reconciliation loop that evaluates metrics every 15 seconds by default. The scaling algorithm applies hysteresis to prevent thrashing: it scales up immediately when thresholds are exceeded but scales down conservatively using a stabilization window. Key tunable parameters include --horizontal-pod-autoscaler-sync-period, --horizontal-pod-autoscaler-downscale-stabilization, and per-HPA behavior policies introduced in Kubernetes 1.18.

• Scale-up policy: Can configure stabilizationWindowSeconds (default 0) and selectPolicy (Max, Min, Disabled)
• Scale-down policy: Default 300-second stabilization prevents premature contraction after transient dips
• Tolerance: 10% tolerance band around target metric prevents unnecessary scaling from minor fluctuations

HPA scaling decisions must coordinate with readiness probes to prevent routing traffic to pods still loading model weights. A newly scaled pod should only report as Ready after completing weight initialization and passing a health check. Without proper probe configuration, HPA can create pods that receive traffic before they can serve predictions, causing HTTP 503 errors and cascading failures under load.

• Startup probe: Introduced in Kubernetes 1.16, allows extended initialization time before liveness/readiness checks begin
• Readiness gate: Custom condition that blocks pod readiness until model weights are fully loaded into GPU memory
• Graceful shutdown: Configure terminationGracePeriodSeconds to allow in-flight inference requests to complete before pod termination during scale-down events

HPA operates at the pod level, but when all nodes are at capacity, new pods remain in Pending state. The Cluster Autoscaler complements HPA by monitoring for unschedulable pods and automatically provisioning additional worker nodes. For GPU-accelerated inference, this requires configuring node selectors, tolerations, and GPU resource requests so the autoscaler provisions nodes with the correct accelerator type.

• Node provisioning latency: Expect 2-5 minutes for new GPU nodes to become available in major cloud providers
• Overprovisioning: Deploy low-priority placeholder pods to pre-warm spare capacity and reduce scheduling latency
• GPU-aware scheduling: Use nvidia.com/gpu resource requests to ensure HPA-scaled pods land on GPU-equipped nodes only

A complementary Kubernetes mechanism that automatically adjusts the CPU and memory reservations of running pods based on historical and current usage. While HPA scales the number of replicas, VPA scales the size of individual pods. For latency-optimized model serving, VPA is critical for right-sizing inference containers—preventing out-of-memory kills during large batch processing while avoiding over-provisioning that wastes cluster resources. VPA can operate in recommend-only mode or automatically update pod specs, though the latter triggers pod restarts which must be managed carefully in production serving paths.

The infrastructure-level scaling controller that provisions or removes worker nodes from the underlying cloud provider when pods fail to schedule due to resource constraints. HPA increases pod count, but if no nodes have available capacity, new pods enter a Pending state. Cluster Autoscaler detects these unschedulable pods and triggers node provisioning. For high-throughput inference spikes, the full chain is: load increases → HPA scales pods → pods can't schedule → Cluster Autoscaler adds nodes. Without this integration, HPA scaling decisions become ineffective at the infrastructure boundary.

The Kubernetes API extension that enables HPA to scale based on application-specific metrics beyond CPU and memory. For model serving, this allows scaling decisions driven by metrics like inference queue depth, P99 latency, or GPU utilization. The custom metrics pipeline requires a metrics aggregator such as Prometheus and an adapter that exposes these values to the HPA controller. This is essential for latency-optimized serving because CPU utilization alone is a lagging indicator—a model may be I/O-bound or waiting on KV cache operations while CPU remains low, delaying necessary scale-out.

A policy that limits the number of voluntary disruptions a replicated application can experience simultaneously. When HPA scales down or Cluster Autoscaler consolidates nodes, pods are terminated. A PDB ensures a minimum number of model serving replicas remain available during these operations using minAvailable or maxUnavailable constraints. Without a properly configured PDB, aggressive scale-down events can drop active inference capacity below the threshold needed to meet latency SLOs, causing request queuing and timeout cascades.

The two Kubernetes resource specifications that directly impact HPA behavior. Requests reserve guaranteed CPU and memory for a pod and are used by the scheduler for placement decisions. Limits cap the maximum resources a pod can consume. HPA calculates target utilization percentages based on requests, not limits. For latency-sensitive model serving, setting requests too low causes HPA to trigger prematurely on minor load increases, while setting them too high wastes cluster capacity. Accurate request sizing—ideally informed by VPA recommendations—is essential for predictable HPA scaling behavior.