Autoscaling

Reactive scaling adjusts resources after a predefined metric threshold is breached. It is the most common autoscaling pattern.

• Mechanism: A cloud monitor (e.g., CloudWatch, Prometheus) tracks metrics like CPU utilization, request queue length, or memory usage. When a metric crosses a threshold (e.g., CPU > 70% for 5 minutes), the autoscaler triggers an action to add or remove instances.
• Use Case: Ideal for workloads with predictable, gradual traffic patterns where a short lag in scaling is acceptable.
• Example: An e-commerce API scales out from 10 to 15 GPU instances during a peak shopping hour based on rising requests-per-second.

Predictive scaling uses historical data and machine learning to proactively adjust capacity before demand changes occur.

• Mechanism: The autoscaler analyzes time-series data to identify recurring patterns (daily, weekly cycles) and forecasts future load. It schedules scaling actions in advance.
• Use Case: Critical for workloads with known, periodic spikes (e.g., business hours, scheduled batch jobs) to eliminate cold start latency for users.
• Example: A video streaming service provisions additional inference servers 30 minutes before the launch of a popular new show, based on viewership patterns.

Autoscaling implements two fundamental architectural approaches to changing capacity.

• Horizontal Scaling (Scale-Out/In): Adds or removes entire instances (nodes) to a cluster. This is the standard method for distributed, stateless services like LLM inference endpoints. It improves fault tolerance and handles traffic spikes effectively.
• Vertical Scaling (Scale-Up/Down): Increases or decreases the compute resources (CPU, GPU, memory) of an existing instance. This is often used for stateful, monolithic applications where distributing load is complex. It typically requires a service restart, causing downtime.

Modern cloud-native LLM serving primarily uses horizontal scaling.

A cooldown period is a configurable wait time after a scaling action during which further scaling activities are suspended.

• Purpose: Prevents rapid, oscillating "flapping" between scale-out and scale-in policies. It allows time for new instances to fully initialize, load the model, and for metrics to stabilize.
• Mechanism: If CPU drops below the scale-in threshold immediately after adding instances, the cooldown timer prevents premature termination before the new capacity is utilized.
• Best Practice: Scale-out cooldowns are typically shorter (e.g., 60 seconds) than scale-in cooldowns (e.g., 300 seconds) to allow for more aggressive growth during surges and conservative contraction.

Autoscaling integrates with health monitoring to ensure only healthy instances serve traffic, maintaining service reliability.

• Health Checks: The load balancer or service mesh performs periodic HTTP or TCP pings on each instance. Failed checks mark an instance as unhealthy.
• Automatic Replacement: The autoscaling group automatically terminates unhealthy instances and launches new ones to maintain the desired capacity, enabling self-healing infrastructure.
• Use Case: Critical for long-running LLM inference servers where GPU memory leaks or software faults can degrade performance over time.

Scheduled scaling allows you to define specific times and dates for capacity changes based on known business events.

• Mechanism: You create scaling actions that trigger at a predetermined time (e.g., Every Monday at 09:00 UTC, set desired capacity to 20). This is a rule-based form of predictive scaling.
• Use Case: Perfect for non-variable, time-based workloads like end-of-month reporting jobs, weekly model retraining pipelines, or supporting fixed business hours for a customer service chatbot.
• Integration: Often used in conjunction with reactive scaling; scheduled scaling sets the baseline capacity, while reactive policies handle unexpected deviations.

LLM inference is fundamentally memory-bound, not CPU-bound. Standard autoscaling metrics like CPU utilization are poor indicators. The primary scaling trigger is KV Cache memory pressure. When the total allocated KV cache across all concurrent requests approaches the GPU's VRAM limit, new requests must wait or be routed to a new instance. Effective autoscalers monitor:

• VRAM Utilization: Percentage of GPU memory consumed by model weights, activations, and the KV cache.
• Request Queue Length: Number of pending inference requests, indicating that existing instances are saturated.
• P95/P99 Latency: A sustained increase in tail latency signals that instances are overloaded and new ones are needed.

The cold start penalty for LLMs is severe, often taking 30-90 seconds to load a multi-billion parameter model into GPU memory. This makes reactive, metric-driven scaling problematic for spiky traffic. Mitigation strategies include:

• Warm Instance Pools: Maintaining a buffer of pre-loaded, idle instances that can immediately accept traffic.
• Predictive Scaling: Using historical traffic patterns (e.g., daily business hour peaks) to proactively scale up before demand arrives.
• Model Caching Tiers: Keeping frequently used models or model variants (e.g., quantized versions) pre-loaded on a subset of instances for faster horizontal scaling.

Throughput and latency are directly governed by dynamic or continuous batching. Autoscaling logic must be aware of the serving engine's batching strategy.

• Optimal Batch Size: Adding more instances does not linearly increase capacity if the per-instance batch size is suboptimal. The autoscaler should consider whether scaling out or increasing the batch size on existing instances is more efficient.
• Heterogeneous Requests: Mixes of short (chat) and long (document summarization) requests affect optimal cluster size. Long-running requests can block batch execution, requiring more instances to maintain latency for short requests.
• GPU Affinity: Some batching strategies work best when a user's session is sticky to a specific instance to leverage cached context. Scaling decisions must consider session affinity to avoid breaking this optimization.

The goal of autoscaling is to minimize total inference cost, which is a function of instance cost and tokens-per-second (TPS) throughput. Naively scaling to minimize latency can lead to exorbitant costs. Effective systems:

• Scale on Cost-Efficiency Metrics: Use metrics like (cost per hour) / (tokens per second) to identify when adding a new instance improves overall cluster efficiency.
• Implement Tiered Scaling: Use a mix of instance types (e.g., cost-optimized for steady state, latency-optimized for peaks) and scale the appropriate tier.
• Leverage Spot/Preemptible Instances: For batch inference workloads or resilient serving backends, use autoscaling groups with spot instances to significantly reduce costs, with warm pools on on-demand instances as a fallback.

LLM applications often involve stateful sessions (e.g., multi-turn conversations where context is cached). Autoscaling must handle this state gracefully.

• KV Cache Persistence: Scaling in (removing an instance) must not drop active sessions. This requires either session draining (finishing all requests) or a distributed KV cache that can be migrated, a complex feature provided by engines like vLLM.
• Stateless Serving Layers: A common pattern is to keep the model serving layer stateless by storing conversation context in an external low-latency database. This simplifies scaling but adds overhead for context retrieval on each turn.

Autoscaling works in concert with load shedding and intelligent request queuing. When a scale-up event is triggered, there is an inherent delay. To maintain service-level objectives (SLOs) during this period:

• Adaptive Queuing: Queue requests with a timeout, rather than immediately rejecting them, to smooth the load while new instances spin up.
• Priority-Based Shedding: Implement request priorities (e.g., premium user, internal batch job). The system can shed low-priority requests during scaling delays to protect high-priority traffic.
• Health Check Damping: Use dampening mechanisms to prevent rapid, oscillating scale-up/scale-down events caused by transient traffic spikes, which is costly and disruptive for LLMs.

The systematic analysis and tuning of software and hardware configurations to maximize the computational efficiency (performance per unit cost) of workloads. For LLMs, this involves techniques like continuous batching, model quantization, and selecting optimal GPU instance types. It is the foundational engineering discipline that makes autoscaling cost-effective by ensuring each provisioned unit delivers maximum throughput.

The process of analyzing workload performance and resource utilization to select the most cost-effective cloud instance type (e.g., GPU model, vCPU count, memory) that meets application requirements without over-provisioning. It is a prerequisite analysis for configuring autoscaling policies, ensuring the cluster scales with appropriately sized nodes.

• Key Activity: Profiling model inference under load to identify the optimal instance family.
• Goal: Avoid paying for unused capacity on oversized instances or encountering throttling on undersized ones.

The latency incurred when a user request triggers the initialization of a new compute instance or container (e.g., loading a model into GPU memory). This results in a significantly slower first response compared to subsequent requests on a warm instance. Autoscaling systems must account for cold start penalties, often by maintaining a warm pool of pre-initialized instances or using predictive scaling to spin up nodes before traffic spikes arrive.

A fault tolerance mechanism where a system under extreme load proactively rejects or delays low-priority requests to prevent catastrophic failure and ensure high-priority requests are served within acceptable latency bounds. While autoscaling aims to add capacity to meet demand, load shedding is the defensive action taken when scaling cannot keep pace or is at its maximum limit, protecting system stability.

A traffic control mechanism that restricts the number of requests a user, API key, or service can make within a specified time window. It protects backend systems, including autoscaling clusters, from being overwhelmed by excessive demand from a single source. Rate limiting works in tandem with autoscaling by smoothing traffic ingress, giving the scaling system time to react to legitimate aggregate demand increases.

An operational framework and cultural practice that brings financial accountability to the variable spend model of the cloud. It enables engineering, finance, and business teams to collaborate on data-driven spending decisions. Autoscaling is a primary technical lever within FinOps, directly linking resource consumption to real-time demand. Effective FinOps practices use autoscaling metrics to forecast costs, allocate budgets, and demonstrate ROI on infrastructure spend.