Autoscaling

Reactive scaling adjusts resources based on real-time, observed metrics from the running system. It is the most common autoscaling pattern.

• Primary Triggers: CPU/GPU utilization, memory pressure, request queue length, and inference latency.
• Scaling Policies: Define rules like "add 2 instances if average GPU utilization > 75% for 5 minutes."
• Cooldown Periods: Mandatory wait times after a scaling action to prevent rapid, costly oscillation and allow metrics to stabilize.
• Example: A sudden viral social media post causes inference requests to spike. A reactive policy scales out GPU instances to handle the load, then scales them in when traffic normalizes.

Predictive scaling uses historical data and machine learning to forecast traffic and provision resources proactively, minimizing cold start latency during anticipated spikes.

• Workload Prediction: Analyzes daily, weekly, or seasonal patterns (e.g., higher usage during business hours, product launch events).
• Proactive Provisioning: Starts instances before the predicted load arrives, ensuring capacity is ready.
• Integration with Forecasting: Often combined with Inference Forecasting models for greater accuracy.
• Example: An e-commerce platform scales up its recommendation model instances before a scheduled flash sale based on traffic patterns from previous sales.

This feature automatically detects and replaces failed or unhealthy instances to maintain service availability and performance, a core aspect of SLA Management.

• Health Checks: Periodic probes (e.g., HTTP /health endpoints) verify an instance can serve requests.
• Automatic Replacement: Unhealthy instances are terminated, and new ones are launched to preserve the desired capacity.
• Graceful Termination: Drains in-flight requests from an instance marked for termination before shutting it down.
• Impact on Cost: Prevents paying for resources that are not contributing to useful work while ensuring SLO Compliance.

Advanced autoscalers incorporate cost metrics into decision logic, optimizing not just for performance but for the Total Cost of Ownership (TCO).

• Mixed Instance Types: Uses a combination of on-demand, Spot Instances, and different GPU generations based on availability and price.
• Scaling for Efficiency: May scale out to smaller, cheaper instances instead of up to a single large one, or vice-versa, based on workload profile.
• Budget Constraints: Can be configured with hard spending limits or Resource Quotas, triggering alerts or blocking scale-out actions.
• Link to TCO: Directly influences operational expenditure, a major component of inference TCO.

Autoscaling is ineffective without seamless integration with traffic routing and cluster management systems.

• Load Balancer Registration: Newly launched instances are automatically registered with a load balancer (e.g., AWS ALB, NGINX) to start receiving traffic.
• Orchestrator Coordination: Works with Inference Orchestrators (e.g., Kserve, Seldon) or Kubernetes to schedule model pods on new nodes.
• Service Discovery: Ensures client requests can find the newly scaled instances.
• Unified Metrics: Scaling decisions often rely on custom metrics exposed by the orchestrator, such as per-model queue depth.

Scheduled scaling allows resources to be scaled at predetermined times, ideal for predictable workload changes and a complement to predictive scaling.

• Fixed-Schedule Actions: "Set desired capacity to 10 instances at 9 AM GMT on weekdays."
• Recurring Patterns: Handles known cycles without requiring continuous metric monitoring.
• Cost Optimization: Used to scale to zero or a minimal baseline during known off-peak periods (e.g., nights, weekends).
• Use Case: A business analytics model used only during office hours can be scaled down to zero overnight, eliminating idle cost.

A granular financial metric that calculates the average expense to generate a single token during LLM inference. It is foundational for cost attribution and chargeback models, allowing teams to precisely track spending against user activity or API calls. Optimizing autoscaling rules directly impacts this metric by reducing idle resource waste during low-traffic periods.

The performance penalty incurred when a new model instance must be initialized from a dormant state, including loading weights into GPU memory. This is a critical trade-off in autoscaling strategies:

• Aggressive scale-down saves cost but increases the risk of cold starts for new requests.
• Warm instance pools maintain readiness but incur higher baseline costs. Engineers balance this latency against the cost of over-provisioning.

The temporary maximum throughput an inference system can handle beyond its sustained baseline. Autoscaling is the primary mechanism to provide burst capacity, but it is constrained by:

• Cloud provider quotas for rapid instance launches.
• Financial budgets for unexpected scaling events.
• Physical hardware availability in the desired region. Planning for burst capacity is essential for handling usage spikes without violating SLA targets.

A defensive strategy where an overloaded system deliberately rejects or delays low-priority requests to maintain stability for high-priority traffic. This is a complementary technique to autoscaling:

• Autoscaling adds resources in response to load.
• Load shedding reduces load when scaling cannot keep pace or is cost-prohibitive. It enforces Quality of Service (QoS) policies and protects core SLO compliance during extreme events.

The practice of selecting cloud compute instances (e.g., GPU type, vCPU count, memory) that precisely match a workload's requirements. Effective autoscaling depends on prior right-sizing:

• Oversized instances lead to low utilization even when scaled, wasting money.
• Undersized instances may scale out excessively, increasing orchestration overhead and network latency. This is a prerequisite for achieving an optimal performance-cost tradeoff on the Pareto frontier.

The central software component that automates the deployment, scaling, and management of model instances. It executes autoscaling policies by:

• Monitoring metrics like request queuing depth and GPU utilization.
• Interacting with cloud APIs to provision or terminate instances.
• Performing cost-aware scheduling across hardware heterogeneous environments (e.g., spot vs. on-demand instances). Tools like Kserve, Ray Serve, and Triton Inference Server include orchestrator capabilities.