Autoscaling

Autoscaling systems monitor live metrics to trigger scaling actions. Key triggers for model serving include:

• CPU/Memory Utilization: Scaling up when average pod usage exceeds a threshold (e.g., 70%).
• Request Queue Length: Adding replicas when pending requests exceed a configured backlog.
• Custom Metrics: Scaling based on application-specific signals like token generation latency or adapter cache hit rate. The scaling policy defines the target value, stabilization windows, and the direction (scale-out or scale-in).

In Kubernetes, Horizontal Pod Autoscaling (HPA) is the primary mechanism for autoscaling stateless workloads like model inference pods. It works by:

• The HPA controller periodically queries the Kubernetes metrics server (or a custom metrics API).
• It calculates the desired replica count using the formula: desiredReplicas = ceil[currentReplicas * (currentMetricValue / desiredMetricValue)].
• It updates the spec.replicas field of the deployment or statefulset object. For PEFT serving, HPA can scale the pool of inference pods that host a base model capable of runtime adapter injection.

The Cluster Autoscaler works in tandem with HPA to add or remove physical nodes from the Kubernetes cluster. Its logic is:

1. When pods cannot be scheduled due to insufficient resources (a "pending" pod), the autoscaler provisions a new node.
2. When nodes are underutilized for a configured period and their pods can be rescheduled elsewhere, the autoscaler safely drains and deletes the node. This ensures the underlying infrastructure scales to meet the resource demands created by HPA's pod scaling decisions, which is critical for GPU-intensive LLM inference.

Autoscaling PEFT deployments introduces unique considerations. A multi-adapter inference server hosts one base model and many small adapter modules (e.g., LoRA). Scaling decisions must account for:

• Adapter Load Distribution: New pods must have access to the same shared adapter registry or cache.
• Cold Start Latency: Scaling out requires loading the base model into memory, which for large models can take minutes. Pre-provisioned warm pods or predictive scaling can mitigate this.
• Metric Selection: Scaling might be driven by aggregate throughput or by per-adapter request rates to ensure quality of service for all tenants.

Beyond reactive scaling, advanced systems use predictive patterns:

• Scheduled Scaling: Based on known traffic patterns (e.g., scaling up before a business hours peak).
• Predictive Scaling: Uses time-series forecasting (e.g., ARIMA, ML models) on historical load metrics to provision resources before demand spikes occur. This is vital for managing the high cost per inference of large models, allowing infrastructure to be scaled in during predictable lulls.

Autoscaling is a runtime component of a full ML pipeline orchestration. It integrates with:

• Model Registry: New model or adapter versions are promoted, triggering deployment updates which the autoscaler then manages.
• CI/CD for ML: Automated canary releases or blue-green deployments involve gradually shifting traffic between autoscaled fleets.
• Monitoring Dashboard: Autoscaling events and metrics (current replicas, CPU) are visualized alongside model performance and drift detection alerts for holistic observability.

Autoscaling from zero replicas introduces a cold start penalty, the delay between a scaling event and a pod being ready to serve. For ML models, this involves:

• Loading large model weights (several GBs) into GPU/CPU memory.
• Running initialization and model compilation steps.
• For PEFT models, dynamically injecting adapter weights via runtime adapter injection. This latency can violate strict latency SLAs for online inference, requiring strategies like maintaining warm pools of pre-loaded replicas.

ML workloads are often stateful, tying compute to specific hardware. Key challenges include:

• GPU Memory Fragmentation: Autoscaling must account for the fixed, large memory footprint of models. A new replica requires a full GPU's memory, not just partial CPU.
• Heterogeneous Instance Types: Scaling may trigger provisioning of different instance types (e.g., A10G vs. H100), affecting performance consistency and cost per inference.
• Adapter Persistence: For multi-adapter inference, the system must ensure the correct adapter weights are loaded on new replicas, complicating stateless horizontal scaling.

Choosing the right metric to trigger scaling is non-trivial. Standard CPU utilization is often a poor proxy for ML load.

• Queue Length: The number of pending requests in the inference server is a direct measure of load but requires custom metric integration.
• GPU Utilization & Memory Pressure: More relevant but can be bursty; sustained high utilization may already indicate latency degradation.
• Token Generation Rate: For LLMs using continuous batching, scaling based on the rate of generated tokens per second can better reflect true computational demand.
• Cost-Aware Scaling: Policies must balance performance against the high cost of GPU instances, potentially implementing scaling delays or aggressive scale-down.

In PEFT deployment scenarios, a single base model serves multiple tenants via different adapters (e.g., LoRA). Autoscaling can cause a thundering herd problem.

• A surge in requests for one tenant's adapter can trigger scale-up, spawning replicas that load all adapters, wasting memory and compute.
• Intelligent routing and adapter versioning are required to ensure replicas scale with granularity, perhaps isolating hot adapters on dedicated replica subsets.
• This complicates load balancing and requires deep integration with the model server (e.g., Triton Inference Server with its ensemble models).

Autoscaling must be coordinated with ML pipeline orchestration and continuous deployment for ML (CD4ML).

• Canary Release & A/B Testing: Traffic splitting for new model versions must work in concert with the autoscaler, which may be provisioning replicas for both canary and stable versions.
• Rollback Speed: A failed deployment must trigger an instant rollback. If the autoscaler has already scaled up the faulty version, it must rapidly terminate those pods and scale the stable version, requiring tight coupling with the model registry.
• State Reconciliation: During a rollback, the system must ensure the correct model binary or adapter weights from the artifact store are loaded on new replicas.

Reactive scaling based on current metrics leads to over-provisioning. Advanced strategies include:

• Predictive Scaling: Using historical request patterns (e.g., daily business cycles) to provision replicas in advance, mitigating cold starts.
• Scale-to-Zero with Async Inference: For batch or non-latency-sensitive workloads, using async inference queues allows scaling compute to zero when idle, dramatically reducing costs.
• Spot Instance Integration: Leveraging interruptible cloud instances (AWS Spot, GCP Preemptible) for scalable, low-cost inference backends, requiring fault-tolerant design and checkpointing for batch inference jobs.

The infrastructure and process of deploying a trained machine learning model to make predictions (inference) available to users or other applications via an API. Autoscaling acts directly on these serving deployments, adjusting resources to meet fluctuating demand.

• Core Function: Hosts the model binary and exposes an inference endpoint.
• Primary Metrics: Latency (response time) and throughput (requests per second).
• Autoscaling Trigger: Metrics like CPU utilization or request queue depth are collected from the serving layer.

A hosted, network-accessible API (e.g., REST or gRPC) that receives input data and returns predictions from a deployed model. This is the entity that client applications call and that autoscaling policies protect from overload.

• Service Contract: Defines the expected input schema and output format.
• SLA Target: Autoscaling helps maintain latency SLAs (e.g., p95 < 200ms) under variable load.
• Scaling Unit: In Kubernetes, an endpoint is typically backed by a scalable Deployment or ReplicaSet of pods.

The number of inference requests a serving system can process successfully per unit of time, measured in requests per second (RPS). Autoscaling aims to maintain target throughput levels as traffic increases.

• Key Performance Indicator: Directly impacts user experience and system capacity.
• Scaling Relationship: To increase aggregate throughput, autoscaling adds more model serving instances (horizontal scaling).
• Batching Impact: Techniques like dynamic batching increase effective throughput per instance, influencing scaling decisions.

An operational metric calculated by dividing the total runtime cost of serving infrastructure (compute, memory) by the number of predictions served. Effective autoscaling minimizes this cost by right-sizing resources.

• Optimization Goal: Run the minimum number of instances required to meet performance SLAs.
• Autoscaling Impact: Prevents over-provisioning (high cost, idle resources) and under-provisioning (failed requests, SLA violations).
• Cloud Billing: Directly tied to the aggregate compute-seconds of provisioned virtual machines or containers.

A serving architecture for PEFT models where a single base model can dynamically load different lightweight adapter modules (e.g., LoRA) per request. This creates unique autoscaling challenges and opportunities.

• Architecture: One shared base model with many small, swappable adapter weights.
• Scaling Complexity: Traffic is per-adapter, not per-base-model. Scaling may need to consider hot/cold adapter caching and memory pressure from many loaded adapters.
• Efficiency: Enables efficient multi-tenant or multi-task serving from a shared resource pool, which autoscaling manages.

A deployment strategy where a new model version is initially rolled out to a small, controlled subset of traffic. Autoscaling is often configured independently for canary and stable deployments during this validation phase.

• Risk Mitigation: Limits impact of a faulty new model.
• Autoscaling Context: The canary deployment typically has its own autoscaling policy, allowing it to handle its allocated traffic slice independently.
• Traffic Splitting: Performed by a service mesh or ingress controller, directing a percentage of requests to the scaled canary pods.