Horizontal Pod Autoscaler (HPA)

HPA scales workloads based on observed metrics, not schedules. The default and most common metric is average CPU utilization. It can also scale based on average memory utilization and, critically, custom and external metrics via the Kubernetes Metrics API. This allows scaling on application-specific KPIs like requests per second, queue length, or any business metric exposed by a custom adapter.

• Core Metrics: CPU, Memory (via resource metrics API).
• Custom/External Metrics: Application-specific metrics (e.g., http_requests_per_second) via the custom metrics API.
• ContainerResource Metrics: Scale based on the resource usage of a specific container within a pod.

Scaling behavior is defined declaratively through a Kubernetes HorizontalPodAutoscaler resource manifest. Engineers specify the target metric (e.g., 70% CPU utilization), the minimum and maximum number of replicas, and optional stabilization windows. The HPA controller continuously works to reconcile the actual state (current metric values) with this declared desired state.

Key configuration fields include:

• scaleTargetRef: The Deployment, StatefulSet, or other scalable resource to manage.
• metrics: List of target metrics and their desired values.
• minReplicas / maxReplicas: The bounds of the scaling range.
• behavior: Configures scaling speed and stabilization to prevent flapping.

To prevent rapid, unnecessary scaling oscillations (thrashing), HPA implements cool-down delays. After a scaling operation, it waits a period (default 5 minutes for upscaling, 3 minutes for downscaling) before evaluating metrics again. The behavior field allows fine-tuning of these stabilization windows and the policies for how many pods can be added or removed per evaluation cycle. This ensures scaling decisions are stable and cost-effective, avoiding reactive scaling to transient traffic spikes.

For modern, microservices-based applications, scaling on CPU is often insufficient. HPA's power is fully realized through integration with the Custom Metrics API and External Metrics API. This requires a metrics adapter like Prometheus Adapter or Datadog Cluster Agent, which translates application-level metrics from monitoring systems into a format HPA can consume. This enables scaling based on:

• QPS (Queries Per Second) for a web service.
• Message backlog in a Kafka consumer.
• Average latency exceeding a threshold.
• Any business logic metric exposed by the application.

HPA scales pods within the resources of a node pool. For true elasticity, it must be paired with the Cluster Autoscaler. If HPA requests new pods but there are insufficient node resources (CPU/memory), the Cluster Autoscaler detects the unschedulable pods and provisions new nodes in the cloud. Conversely, if nodes become underutilized after HPA scales down, the Cluster Autoscaler can remove nodes to reduce infrastructure costs. This creates a fully automated, two-layer scaling system: HPA for application pods, Cluster Autoscaler for infrastructure nodes.

The HPA operates on a continuous control loop. It queries the Kubernetes Metrics Server or a custom metrics API (like Prometheus Adapter) at a default interval of 15 seconds to collect metrics for targeted resources. It then compares the current metric value (e.g., average CPU utilization across all pods) against the target value defined in the HPA specification. Using a simple formula:

desiredReplicas = ceil[currentReplicas * (currentMetricValue / desiredMetricValue)]

It calculates the desired number of pods and instructs the deployment or replica set controller to scale accordingly. This ensures resource consumption aligns dynamically with application load.

CPU utilization is often insufficient for scaling LLM inference workloads, which are constrained by GPU memory and throughput. Effective HPA configuration for LLMs relies on custom metrics such as:

• Request Queue Length: The number of inference requests waiting in a queue.
• Average Token Generation Latency: The time taken per output token, which increases under load.
• GPU Memory Utilization: Percentage of GPU VRAM in use.
• Concurrent Requests per Pod: The number of active requests a pod is handling.

These metrics are exposed by inference servers like vLLM or TGI and collected via Prometheus. An HPA can be configured to scale based on a target average queue length (e.g., scale up if queue length > 5) to maintain low latency.

To prevent rapid, flapping scaling actions, HPA provides stabilization window controls:

• scaleUpStabilizationWindowSeconds: Minimum time the metric must indicate a need to scale up before acting (default 0).
• scaleDownStabilizationWindowSeconds: Minimum time the metric must indicate a need to scale down before acting (default 300 seconds/5 minutes).

Policies define how many pods can be added or removed in a single action:

• policies: [{type: Pods, value: 4, periodSeconds: 60}] limits scaling to 4 pods per minute. For LLMs, a conservative scale-down policy is crucial because pod startup (loading a multi-GB model) can take minutes. Aggressive scale-down can lead to thrashing.

HPA scales pods within the constraints of available cluster nodes. The Cluster Autoscaler (CA) complements HPA by automatically adjusting the number of nodes in the node pool. The workflow is:

1. HPA requests new pods due to high load.
2. If there are insufficient resources on existing nodes, the new pods enter a Pending state.
3. The Cluster Autoscaler detects pending pods and provisions a new node in the cloud provider.
4. Once the node is ready, the pods are scheduled and start.

For GPU-based LLM pods, this requires node pools with the appropriate accelerator type. The CA's provisioning time (often 1-3 minutes for GPU nodes) is a key factor in overall scaling latency.

Scaling stateful LLM inference presents unique challenges:

• Cold Start Latency: Loading a 10B+ parameter model into GPU memory can take 30-60 seconds. HPA scaling events directly impact end-user latency.
• Pod Resource Requests: LLM pods must define precise resources.requests for GPU (nvidia.com/gpu), CPU, and memory. HPA cannot scale pods that cannot be scheduled.
• Multi-Model Endpoints: A single deployment may host multiple model adapters. Scaling metrics must aggregate load across all served models.

A common pattern is to use HPA with a minimum replica count > 0 (e.g., 2) to maintain a warm pool of ready pods, absorbing traffic spikes while slower scale-out occurs.

For business-level scaling, HPA can use External Metrics sourced from systems outside Kubernetes. For example, scaling based on:

• Messages in a Kafka topic (e.g., pending inference jobs).
• Custom business logic metrics from an application.

This is typically implemented using the Prometheus Adapter, which translates Prometheus queries into metrics the Kubernetes custom metrics API understands. A sample HPA spec for an external metric:

yamlCopy
metrics:
- type: External
  external:
    metric:
      name: queue_messages_ready
    target:
      type: AverageValue
      averageValue: 10

This would scale the deployment to maintain an average of 10 ready messages per pod.

A Kubernetes controller that automatically adjusts the CPU and memory resource requests and limits for pods based on historical usage, rather than scaling the number of pod replicas. It is complementary to HPA:

• Right-sizes resources: Prevents over-provisioning and under-provisioning at the container level.
• Memory-based scaling: Can scale based on memory usage, where HPA's default support is more limited.
• Usage: Often used for stateful workloads where horizontal scaling is complex, or in conjunction with HPA for comprehensive resource optimization.

A Kubernetes component that automatically adjusts the size of the node pool in a cluster. It works in tandem with HPA and VPA:

• Node-level scaling: Adds or removes worker nodes from the cluster based on pending pods that cannot be scheduled due to resource constraints.
• Reacts to pod scaling: When HPA creates new pods that require capacity, the Cluster Autoscaler provisions new nodes to host them.
• Cost optimization: Scales down nodes when they are underutilized, reducing cloud infrastructure costs.

A Kubernetes extension API that allows the HPA to scale based on application-specific metrics beyond default CPU and memory. This is critical for scaling modern, metric-rich applications:

• Beyond CPU: Enables scaling based on queries per second (QPS), queue length, application latency, or business logic metrics (e.g., orders per minute).
• Integration: Requires a metrics adapter (like Prometheus Adapter) to translate metrics from monitoring systems (Prometheus, Datadog) into the Kubernetes API.
• Declarative scaling: Pods can be scaled based on the exact metric that indicates true load for the application.

A Kubernetes policy that limits the number of concurrent voluntary disruptions to a set of pods. It is crucial for maintaining availability during operations that involve HPA and cluster changes:

• Voluntary disruptions: Includes actions initiated by the cluster, like node drain for scaling down, or rolling updates.
• Protects during scaling: When Cluster Autoscaler removes a node, PDBs ensure a minimum number of application pods remain available.
• Policy types: Defines minAvailable (e.g., "at least 3 pods must run") or maxUnavailable (e.g., "at most 1 pod can be down") for an application.

A dedicated infrastructure layer that provides advanced traffic management and rich telemetry, which can be used as a source for HPA custom metrics:

• Rich application metrics: Provides out-of-the-box metrics like request rate, error rate, and latency (p95, p99) between services.
• Traffic-based scaling: HPA can be configured to scale deployments based on the request rate or latency metrics emitted by the service mesh (e.g., Istio, Linkerd).
• Canary analysis integration: Enables sophisticated progressive delivery where traffic splitting and autoscaling work in concert for safe rollouts.