Cold Start

This is the most common trigger in cloud-native environments. To minimize costs, inference endpoints are configured to scale to zero replicas when no traffic is received for a defined period. When a new request arrives, the orchestrator (e.g., Kubernetes) must:

• Schedule a new pod on a node.
• Pull the container image.
• Load the base model weights and any adapter modules into GPU memory.
• Execute model warm-up inferences. This entire sequence, which can take tens of seconds for large models, is the cold start penalty paid for perfect cost efficiency during idle periods.

The core computational delay. Loading a multi-gigabyte model from persistent storage (e.g., network-attached disk, object store) into GPU VRAM is I/O and memory-bandwidth intensive. For PEFT servers, this involves two stages:

1. Loading the frozen base model (e.g., Llama 3 70B).
2. Loading and integrating the task-specific LoRA weights or adapter files. Complexities like decompressing quantized models (e.g., QLoRA) or merging adapters into base weights add significant CPU-bound time before the first inference can be scheduled on the GPU.

Architectures designed for efficiency create cold start complexity. In multi-adapter serving, a single base model hosts many adapters. If an adapter for a specific task or tenant is not cached in memory, a cache miss occurs. Serving the request then requires:

• Locating the adapter weights in storage.
• Loading them into the GPU's memory space.
• Configuring the model's execution graph to use the new adapter. This adapter switching latency is a form of cold start specific to dynamic PEFT systems, where the penalty scales with the number of inactive adapters.

Before software can start, hardware must be ready. In cloud environments, scaling from zero may require provisioning a new virtual machine or GPU instance. This involves:

• The cloud provider's scheduler finding available hardware.
• Booting the OS and attaching volumes.
• Installing necessary drivers (e.g., NVIDIA CUDA). This underlying infrastructure spin-up can add unpredictable latency, often ranging from 30 seconds to several minutes, depending on GPU scarcity and region.

The software environment itself must be initialized. A containerized inference server (e.g., Triton Inference Server, vLLM) must:

• Start the Python runtime and import all libraries.
• Initialize the inference framework and load its internal schedulers.
• Establish connections to auxiliary services (e.g., telemetry endpoints, model registries, distributed tracing collectors).
• Bind to the network port and pass readiness health checks. This boot sequence, while faster than model loading, is a fixed cost that contributes to the overall cold start duration.

Autoregressive models rely on the Key-Value (KV) Cache to avoid recomputing attention for previous tokens. During a cold start, this cache is empty. The first request cannot benefit from the performance optimization the cache provides for subsequent tokens in a sequence. While the cache warms up quickly per request, systems serving very long contexts (e.g., 128K tokens) will see a more pronounced initial compute cost for the first request as the entire attention context is computed without a cache.

Maintaining a small, always-on pool of ready model instances to absorb initial traffic spikes, preventing the need for scaling from zero. This is often managed alongside autoscaling with a minimum replica count.

• Minimum Replicas: Setting a minReplicas > 0 in a Kubernetes Horizontal Pod Autoscaler (HPA) configuration.
• Scale-to-Zero Override: Used for development or low-traffic endpoints, but often disabled in production for latency-critical applications.
• Cost-Latency Trade-off: Balances infrastructure cost against the guarantee of sub-second p95 latency.

Reducing the time to fetch and deserialize the model artifact itself. For PEFT methods like LoRA or Adapters, this involves strategies to avoid loading the full base model repeatedly.

• Merged Weights for Inference: Pre-merge trained LoRA weights with the base model to create a single, standard model file, eliminating runtime merge overhead.
• Multi-Adapter Serving: Using frameworks that keep the base model in memory and dynamically load small adapter weights (<100MB) via adapter switching, which is faster than loading a full multi-billion parameter model.
• Model Caching: Staging model files on fast, local NVMe disks or in-memory filesystems instead of pulling from remote object storage (e.g., S3) on every cold start.

Using historical traffic patterns and machine learning to forecast demand and scale resources preemptively, before the load arrives.

• Schedule-Based Scaling: Kubernetes CronHPA or KEDA scalers to add replicas before predictable daily traffic peaks.
• ML-Driven Forecasting: Analyzing metrics to predict surges and trigger scaling actions minutes in advance.
• Queue-Based Scaling: Monitoring inference request queue depth and scaling up when the queue starts to build, providing a buffer before latency degrades.

Techniques to make the initial loading phase faster or to start serving requests before the model is fully optimized.

• Skeleton Loading: Loading the model architecture and a subset of critical layers first to accept requests, while streaming remaining parameters in the background (advanced technique).
• Quantized Initial Load: Initially loading a quantized (e.g., INT8) version of the model for faster memory transfer, then swapping in higher-precision weights.
• Just-In-Time Layer Loading: Used in some research systems, where layers are loaded from disk only as needed for the first few requests.

Model warm-up is the proactive process of loading a machine learning model and its dependencies into memory and executing initial, dummy inferences before the endpoint receives live production traffic. This process initializes the model's computational graph, populates caches (like the KV Cache), and allows the just-in-time (JIT) compiler to optimize kernels, ensuring the first real user request does not incur the full latency penalty of a cold start. It is a standard mitigation strategy employed during deployment or scaling events.

Autoscaling is a cloud infrastructure capability that automatically adjusts the number of active compute instances (e.g., pods, VMs) based on real-time demand metrics like CPU utilization, memory pressure, or request queue length. It is a primary cause of cold start latency: when demand drops to zero, services scale in to save cost; a subsequent traffic spike forces a scale-out event, requiring new instances to be provisioned and models to be loaded from scratch. Strategies like predictive scaling or maintaining a minimum number of warm instances are used to balance cost and latency.

An inference server (e.g., Triton Inference Server, vLLM, TGI) is specialized software designed to host and serve machine learning models via network APIs. It manages the entire inference lifecycle, which directly impacts cold start characteristics. Key responsibilities include:

• Loading and initializing model artifacts from disk.
• Managing dynamic batching and continuous batching queues.
• Handling hardware acceleration (GPU/CPU).
• The server's efficiency in loading models, managing memory, and initializing frameworks is a major determinant of cold start duration.

Multi-adapter serving is an architecture where a single instance of a large base model (e.g., Llama 3, GPT) can dynamically load and switch between multiple, smaller adapter or LoRA modules fine-tuned for different tasks or tenants. This presents a unique cold start challenge: while the base model may remain warm in memory, loading a specific adapter module for the first time requires reading its weights from storage and integrating them into the computational graph, causing a per-adapter cold start. Efficient caching and pre-loading strategies are critical for performance.

The Key-Value (KV) Cache is a critical memory buffer in transformer-based autoregressive models. During text generation, it stores the computed key and value tensors for all previous tokens in a sequence, preventing their recomputation for each new token. Initializing this cache is a significant component of cold start latency for LLMs. The first request must compute these tensors from scratch, while subsequent requests benefit from the cached computations. Optimization techniques like PagedAttention (in vLLM) are designed to manage this cache more efficiently during both startup and operation.

A health check is a periodic probe (e.g., an HTTP /health endpoint) used by container orchestrators like Kubernetes and load balancers to determine if a service instance is operational and ready to receive traffic. For inference endpoints, a successful health check typically confirms the model is loaded and the server is initialized. Cold start is the period between when a new instance is scheduled and when it passes its first health check. Implementing a readiness probe that only succeeds after model warm-up is complete ensures traffic is not routed to a partially initialized server, which would result in high latency or errors.