Horizontal Pod Autoscaler (HPA)

HPA scales workloads based on real-time metrics. While originally designed for CPU and memory utilization, modern HPAs support custom and external metrics via the Kubernetes Metrics API.

• Core Metrics: Scale based on average CPU or memory usage across all pods.
• Custom Metrics: Scale using application-specific metrics like HTTP requests per second or queue length, published by the application.
• External Metrics: Scale based on metrics from outside the Kubernetes cluster, such as cloud provider queue depth or a custom monitoring system.

This allows scaling decisions to be directly tied to business logic, not just resource consumption.

For advanced use cases beyond CPU/memory, HPA relies on the Kubernetes Custom Metrics API and External Metrics API. These APIs are typically provided by monitoring adapters like Prometheus Adapter.

• Prometheus Adapter: Queries a Prometheus time-series database, converts PromQL queries into metrics the HPA can understand, and serves them via the Custom Metrics API.
• Workflow: Application emits metrics -> Prometheus scrapes them -> Prometheus Adapter exposes them -> HPA reads them and makes scaling decisions.

This architecture enables powerful scaling rules, such as scaling a model inference deployment based on the average request latency or the number of pending requests in a queue.

The most direct scaling strategy for an LLM API is based on incoming request volume. HPA can scale pods based on a custom metric like Queries Per Second (QPS) or Requests Per Second (RPS).

• How it works: A metrics adapter (like Prometheus Adapter) exposes the current request rate from the API gateway or service mesh as a custom metric. HPA uses this to add replicas as traffic spikes.
• Example Target: Scale up when the average QPS per pod exceeds 10, and scale down when it falls below 2.
• Consideration: Pure request rate doesn't account for request complexity (input/output token count), so it should be combined with other metrics for optimal resource use.

For GPU-accelerated inference, scaling based on GPU memory utilization or GPU compute utilization is essential, as this is typically the bottleneck resource.

• How it works: Using tools like the NVIDIA GPU Operator or DCGM Exporter, GPU metrics are collected and made available to the Kubernetes metrics pipeline. HPA can target a metric like DCGM_FI_DEV_GPU_UTIL.
• Key Metric: GPU Memory Utilization (DCGM_FI_DEV_FB_USED) is often more stable and indicative of model load than compute utilization, which can spike briefly per token.
• Challenge: GPU metrics can be 'noisy' during autoregressive generation. Using an averaged metric over a longer period (e.g., 2-3 minutes) helps prevent rapid, unnecessary scaling fluctuations.

The Vertical Pod Autoscaler (VPA) automatically adjusts the CPU and memory resource requests and limits of pods within a deployment. Unlike HPA, which scales horizontally by adding pod replicas, VPA scales vertically by modifying the resource specifications of existing pods.

• Use Case: Corrects misconfigured resource requests, preventing out-of-memory (OOM) kills or wasted capacity.
• Limitation: Often requires pods to restart to apply new resource limits, which can cause brief service disruption.
• Combination with HPA: Can be used alongside HPA for comprehensive scaling, but requires careful configuration to avoid conflicts over the same resource metrics.

The Cluster Autoscaler (CA) is a Kubernetes component that automatically adjusts the size of the node pool in a cluster. It scales the infrastructure layer by adding or removing worker nodes based on pending pods that cannot be scheduled due to resource constraints.

• Trigger: Acts when pods fail to schedule (are "pending") due to insufficient CPU, memory, or other node resources.
• Synergy with HPA: Works downstream from HPA. HPA creates more pods; if the cluster lacks capacity, CA provisions new nodes to host them.
• Cloud Provider Integration: Relies on cloud provider APIs (e.g., AWS Autoscaling Groups, GCP Managed Instance Groups) to manage node instances.

The Custom Metrics API (part of the Kubernetes Metrics Server ecosystem) extends HPA beyond core CPU and memory usage. It allows HPA to scale based on application-specific metrics like HTTP requests per second, queue length, or business logic metrics (e.g., orders per minute).

• Architecture: Requires an additional adapter (e.g., Prometheus Adapter) that fetches metrics from a monitoring system like Prometheus and exposes them to the Kubernetes API.
• Metric Types: Supports both per-pod and object metrics. For example, scaling based on the depth of a named queue in a messaging system.
• Declaration: In the HPA spec, you reference these custom metrics using the metrics field with a type: Pods or type: Object specification.

The Kubernetes Metrics Server is a core cluster add-on that collects resource usage data (CPU and memory) from each node's Kubelet and exposes it via the Kubernetes Metrics API. It is the foundational data source for the HPA's default scaling decisions.

• Function: Aggregates resource usage metrics; does not store historical data.
• Dependency: HPA's type: Resource scaling (for CPU/memory) requires the Metrics Server to be installed and running.
• Difference from Monitoring Tools: It is designed for automatic scaling, not for long-term monitoring or alerting. Tools like Prometheus are used for those purposes and can feed the Custom Metrics API.

A Pod Disruption Budget (PDB) is a Kubernetes resource that limits the number of concurrent voluntary disruptions of pods belonging to an application. It is critical for maintaining availability during operations triggered by HPA (scale-down) or cluster maintenance.

• Purpose: Ensures a minimum number or percentage of pods for a deployment/statefulset remain available during voluntary disruptions.
• Interaction with HPA: When HPA decides to terminate pods during a scale-in event, it respects the PDB. If scaling down would violate the PDB, the termination is postponed.
• Configuration: Defined using minAvailable or maxUnavailable fields, applied to a set of pods via label selectors.

The Kubernetes Operator Pattern is a method of extending Kubernetes to manage complex, stateful applications using custom controllers and Custom Resource Definitions (CRDs). Advanced scaling logic for stateful services (like databases) is often implemented via Operators rather than basic HPA.

• Beyond Stateless HPA: Operators can encode domain-specific knowledge for safe scaling of stateful workloads, handling tasks like re-sharding, data replication, and backup coordination during scale events.
• Custom Resource (CR): An Operator might define a DatabaseCluster CRD. Scaling would be managed by updating the replica field in this custom resource, which the Operator's controller reconciles.
• Example: The etcd Operator, PostgreSQL Operator, or Prometheus Operator manage their own scaling lifecycles with more sophistication than HPA provides.