Autoscaling

Autoscaling decisions are triggered by real-time metrics, not schedules. The most common scaling signal is average CPU utilization, but modern systems use richer metrics.

• Custom Metrics: Scaling can be based on application-level metrics like request queue length, inference latency percentiles, or tokens generated per second.
• Multi-Metric Policies: Advanced policies require multiple conditions (e.g., high CPU and high memory) to trigger a scale-out, preventing flapping.
• Prometheus Integration: In Kubernetes, the Horizontal Pod Autoscaler (HPA) can query custom metrics from Prometheus, allowing scaling based on model-specific load.

Autoscaling primarily refers to horizontal scaling (scale-out/in), which changes the number of identical service instances (pods). This contrasts with vertical scaling (scale-up/down), which changes the resource allocation (CPU/RAM) of a single instance.

• Horizontal Scaling: Adds or removes pods in a deployment. Ideal for stateless services like inference servers, as it provides high availability and handles traffic spikes.
• Vertical Scaling: Requires pod restarts to change resource limits, causing brief downtime. Less common for dynamic autoscaling but can be used for stateful workloads.
• Cluster Autoscaler: Works in tandem with horizontal scaling by automatically adding or removing nodes from the Kubernetes cluster to accommodate the scheduled pods.

To prevent rapid, costly oscillation between scaling actions, autoscalers implement stabilization windows.

• Scale-Up Cool-Down: A mandatory wait period after adding pods before another scale-up can occur, allowing time for new pods to start and receive traffic.
• Scale-Down Cool-Down: A longer delay before removing pods, ensuring a transient drop in load doesn't lead to premature scaling in. The Kubernetes HPA default scale-down delay is 5 minutes.
• Pod Disruption Budgets (PDBs): Define the minimum number of available pods during voluntary disruptions, which the autoscaler respects when scaling in, ensuring application availability.

For cost optimization, services with intermittent traffic can be configured to scale to zero replicas when idle, eliminating all running costs.

• Knative / KEDA: Platforms like Knative and Kubernetes Event-Driven Autoscaling (KEDA) enable scaling from zero based on event sources (e.g., a message queue length) or HTTP request arrival.
• Cold Start Penalty: The primary trade-off is latency. A request that triggers a scale-from-zero must wait for the pod to be scheduled, the container image to be pulled, and the model warm-up process to complete.
• Use Case: Ideal for development endpoints, batch inference jobs, or low-traffic internal APIs where occasional high latency is acceptable for significant cost savings.

Beyond reactive scaling based on current metrics, systems can use predictive or scheduled policies.

• Scheduled Scaling: Rules scale resources up before a known peak (e.g., business hours) and down afterward. This is simple but inflexible.
• Predictive Scaling: Uses historical metric data and machine learning to forecast future load and proactively scale resources. For example, it can anticipate a daily traffic surge 30 minutes before it occurs, mitigating cold start issues.
• Hybrid Approaches: A common pattern uses predictive scaling for baseline capacity and metric-driven scaling to handle unexpected deviations from the forecast.

Autoscaling interacts directly with core inference server optimizations, requiring careful configuration.

• Dynamic Batching: Effective batching requires a steady stream of requests. Over-aggressive scaling-in can reduce request density, harming batch efficiency and increasing latency.
• Continuous Batching: More resilient to variable load, as new requests can join a running batch. Autoscaling policies must account for the memory footprint of the KV Cache held by ongoing generations.
• Multi-Adapter Serving: In a system serving multiple LoRA adapters, scaling decisions must consider the memory overhead of keeping multiple adapter sets loaded in a pod's memory, influencing the pod's resource requests and limits.

The Horizontal Pod Autoscaler (HPA) is the native Kubernetes controller that implements autoscaling for containerized applications. It automatically adjusts the number of pod replicas in a deployment based on observed metrics like CPU, memory, or custom Prometheus metrics.

• Core Mechanism: Continuously monitors target metrics against defined thresholds.
• Scaling Triggers: Scales out (adds pods) when metrics exceed a target; scales in (removes pods) when utilization is low.
• Use Case for Inference: Essential for scaling PEFT inference servers (e.g., vLLM, TGI) based on request per second (RPS) or GPU memory pressure.

Cold start is the latency penalty incurred when a service instance must be initialized from scratch because it is not already loaded in memory. In inference serving, this occurs when autoscaling spins up a new pod to handle increased load.

• Primary Impact: The first requests to a new instance experience high latency as the model loads, weights load into GPU memory, and the server initializes.
• Mitigation Strategies: Use model warm-up scripts, maintain a minimum replica count, or employ predictive scaling based on traffic patterns.
• PEFT Consideration: For multi-adapter serving, cold starts may also involve loading the specific adapter weights required for the request.

Model warm-up is the proactive process of loading a machine learning model into memory and performing initial dummy inferences before it receives live production traffic.

• Purpose: Eliminates the cold start penalty for the first real user requests by ensuring the model graph is compiled, weights are cached, and GPU kernels are primed.
• Implementation: Often executed as an initContainer or startup probe in a Kubernetes pod definition, sending sample requests to the local inference endpoint.
• Critical for Autoscaling: A key practice to ensure new pods brought online by autoscalers are immediately ready to serve with target latency.

Observability is the measure of how well the internal states of a system can be inferred from its external outputs. For autoscaling systems, it is the foundation for making intelligent scaling decisions.

• Three Pillars: Relies on metrics (e.g., CPU, GPU util, queue length), logs (request/error logs), and traces (distributed tracing of requests).
• Autoscaling Dependency: Autoscalers like HPA consume these observable metrics to trigger scaling events. Without granular metrics, scaling is ineffective.
• Key Metrics for Inference: Request latency (p95/p99), tokens per second, GPU memory utilization, and error rates are essential for tuning autoscaling rules.

Multi-tenancy is an architecture where a single instance of a software application serves multiple isolated customer organizations (tenants). In PEFT serving, this often maps to a single base model hosting multiple adapters.

• Autoscaling Challenge: Traffic patterns can vary significantly per tenant. Autoscaling must consider aggregate load while ensuring performance isolation.
• Scaling Strategies: May involve per-tenant queue management or scaling based on a blend of global and tenant-specific metrics.
• Resource Efficiency: Enables efficient resource sharing via autoscaling, as a single model pool can dynamically scale to serve many tenants.

Rate limiting is a control mechanism that restricts the number of requests a client or tenant can make to an API within a specified time window. It works in tandem with autoscaling to protect system stability.

• Defensive Role: Prevents a single tenant or misbehaving client from overwhelming newly scaled resources, allowing autoscalers time to react.
• Implementation Layer: Often applied at the API gateway or ingress controller before requests reach the scalable inference service.
• Complement to Autoscaling: While autoscaling adds capacity, rate limiting ensures fair usage and prevents resource exhaustion, creating a stable serving environment.