Horizontal Pod Autoscaling (HPA)

The most common scaling trigger is based on CPU and memory utilization. HPA calculates the current utilization as a percentage of the requested resources defined in the pod's resource requests. For example, if a pod requests 500m CPU and is using 250m, utilization is 50%. The HPA controller uses this to determine if the average across all pods exceeds or falls below the target threshold, triggering a scale-out or scale-in event.

Beyond CPU and memory, HPA can scale based on custom metrics (application-specific, e.g., requests per second, queue length) and external metrics (from systems outside the cluster, e.g., cloud pub/sub queue depth). This requires a metrics server for core resources and an adapter for the Kubernetes Metrics API, such as the Prometheus Adapter, to expose custom metrics. This allows scaling on business logic, like the number of pending tasks in a job queue.

HPA calculates the desired replica count using the formula: desiredReplicas = ceil[currentReplicas * (currentMetricValue / desiredMetricValue)]. To prevent rapid, flapping scale changes, HPA employs cooldown delays:

• scaleUp and scaleDown stabilization windows (configurable, defaults vary).
• The --horizontal-pod-autoscaler-downscale-stabilization flag (default 5 minutes) prevents scaling down too aggressively after a recent scale-up.

The behavior field in the HPA spec (API version autoscaling/v2) allows fine-grained control over scaling speed and direction. Policies can be set independently for scaleUp and scaleDown:

• policies: Define the percentage or number of pods that can be added/removed per period.
• selectPolicy: Can be Min, Max, or Disabled to control which policy is used.
• stabilizationWindowSeconds: Prevents flapping by considering historical metric values.

HPA works in tandem with the Cluster Autoscaler. While HPA adjusts the number of pods, the Cluster Autoscaler adjusts the number of nodes in the cluster. If HPA scales out pods and there are insufficient node resources, the Cluster Autoscaler will provision a new node to accommodate the pending pods. Conversely, it can remove underutilized nodes after HPA scales in pods. This creates a fully elastic infrastructure layer.

In Edge AI deployments, HPA manages the scale of inference server pods hosting ML models. It can scale based on:

• Custom metrics like inference request latency or batch queue size.
• External metrics from edge device fleets. This ensures edge clusters handle sporadic inference loads efficiently, scaling model replicas across available edge nodes to meet low-latency service-level objectives while conserving resources during low-activity periods.

The Kubernetes Metrics Server is the foundational component for HPA. It is a cluster-wide aggregator of resource usage data, collecting metrics like CPU and memory utilization from each node's Kubelet. HPA queries the Metrics Server API to obtain the current resource consumption of pods. Without it, HPA cannot function for standard resource metrics. It is typically deployed as a single pod in the kube-system namespace and is a prerequisite for any basic HPA configuration.

To scale on custom metrics (e.g., requests per second, queue length), HPA requires an external metrics pipeline. The Prometheus Adapter is the standard solution. It acts as a bridge between HPA and a Prometheus monitoring stack.

• It translates Prometheus queries into the custom metrics API that HPA understands.
• Allows scaling based on virtually any metric stored in Prometheus.
• Configuration involves writing PromQL queries that the adapter will execute at a regular interval to feed metrics to the HPA controller.

Vertical Pod Autoscaling (VPA) is a complementary technology to HPA. While HPA adjusts the number of pod replicas, VPA automatically adjusts the resource requests and limits (CPU, memory) of individual pods.

• Use Case: Corrects misconfigured resource requests. A pod under-provisioned at startup can be given more resources without a restart (with the UpdateMode: Initial).
• Caution: VPA in UpdateMode: Auto typically requires pod recreation, which can be disruptive. It is often used in conjunction with HPA for memory-bound applications, where HPA handles request load and VPA manages per-pod memory allocation.

The Cluster Autoscaler works in tandem with HPA at the infrastructure layer. When HPA scales out a deployment and there are insufficient cluster resources to schedule the new pods, the pods enter a Pending state.

• The Cluster Autoscaler detects these unschedulable pods.
• It automatically provisions new worker nodes (in cloud environments) to accommodate the workload.
• Conversely, it scales down nodes that are underutilized, consolidating pods to reduce costs.
• This creates a full-stack, elastic scaling solution: HPA scales pods, CA scales the underlying nodes.

Service meshes like Istio provide rich application-layer traffic metrics (e.g., requests per second, request duration, error rates) that are ideal for autoscaling. The integration flow is:

1. Istio's telemetry system (Mixer/Telemetry V2) collects metrics from the service proxy (Envoy).
2. These metrics are exposed to Prometheus.
3. The Prometheus Adapter is configured with rules to make these metrics available to the Kubernetes custom metrics API.
4. HPA is configured to scale based on metrics like istio_requests_per_second. This enables scaling directly on business-relevant traffic patterns rather than just infrastructure utilization.

A Pod is the smallest and simplest deployable unit in Kubernetes. It represents a single instance of a running process in a cluster and can contain one or more application containers that share:

• Storage volumes
• A unique network IP
• Configuration for how to run the containers

For edge AI, a Pod typically packages a model inference server, its dependencies, and any sidecar containers for logging or monitoring. HPA scales the number of identical Pod replicas in a Deployment or StatefulSet.

Vertical Pod Autoscaling (VPA) is a Kubernetes component that automatically adjusts the CPU and memory resource requests and limits for Pods based on historical usage. Unlike HPA, which changes the number of Pods, VPA changes the size of individual Pods.

Key differences from HPA:

• Scaling Dimension: Modifies pod resource allocation (vertical) vs. pod count (horizontal).
• Use Case: Optimizes resource utilization for applications with unpredictable but stable growth, rather than handling bursty traffic.
• Edge Consideration: On resource-constrained edge nodes, VPA can help right-size models to fit within available memory, but may require pod restarts.

The Cluster Autoscaler is a Kubernetes component that automatically adjusts the size of the node pool in a cluster. It adds nodes when Pods fail to schedule due to insufficient resources and removes nodes that are underutilized.

Interaction with HPA:

1. HPA creates new Pod replicas to handle load.
2. If there is no capacity on existing nodes for the new Pods, they become "pending."
3. The Cluster Autoscaler detects pending Pods and provisions a new node.
4. The Pods then schedule onto the new node.

For edge deployments, this is often less relevant as edge node fleets are typically static, but it's critical in cloud-based management clusters.

The Custom Metrics API (part of the Kubernetes Metrics Server ecosystem) allows HPA to scale based on application-specific metrics beyond standard CPU and memory. This is essential for scaling AI/ML workloads where load is better measured by:

• Inference requests per second (RPS)
• Queue length of a processing pipeline
• GPU utilization percentage
• Application-level business metrics (e.g., predictions served)

To use it, a metrics adapter (like Prometheus Adapter) must be installed to serve custom metrics from a monitoring system to the Kubernetes API.

A Kubernetes Operator is a method of packaging, deploying, and managing a Kubernetes application using custom resources and controllers. Operators encode human operational knowledge (like backup procedures or complex scaling logic) into software.

Relation to HPA: For stateful edge AI applications (e.g., a distributed model training job or a database-backed inference service), a custom Operator might manage scaling logic more sophisticated than HPA's simple metrics, coordinating with StatefulSets and persistent volumes. The Operator can use the HPA API or implement its own control loop.

Desired State is a core Kubernetes paradigm where a user declares the intended configuration of the system (e.g., "run 5 replicas of this Pod"). Controllers like the HPA continuously observe the actual state and make changes to drive the system toward the desired state.

HPA's Role:

1. User defines a desired metric target (e.g., 70% CPU utilization).
2. HPA's control loop constantly checks the actual metric value.
3. If actual > desired, HPA updates the Deployment's desired replica count upward.
4. The Deployment controller then works to create more Pods to match this new desired state.

This reconciliation loop is fundamental to all automated orchestration in Kubernetes.