Cold Start Latency

This is the foundational delay where the model's weights and architecture are read from persistent storage (e.g., disk, network storage) into host memory. For large models, this involves:

• Deserializing a multi-gigabyte file (e.g., SafeTensors, PyTorch .pt).
• Transferring the weights across the storage bus (PCIe/NVMe).
• The duration scales linearly with model size and is heavily dependent on storage I/O speed and filesystem caching.

Before computation can begin, the serving system must prepare the execution environment. This involves:

• GPU Context Creation: Initializing the CUDA/ROCm driver context, which has a fixed overhead.
• Kernel Compilation (JIT): For frameworks like PyTorch, the first execution triggers Just-In-Time compilation of model operators into optimized GPU kernels, causing a one-time delay. Engines like TensorRT or ONNX Runtime perform this step ahead-of-time (AOT).
• Memory Allocation: Reserving device memory for model weights and runtime buffers.

To ensure the first real user request isn't penalized, systems often execute warm-up requests. This phase serves two critical purposes:

• Populating Caches: Fills CPU/GPU memory caches with model weights, reducing access latency for subsequent inferences.
• Stabilizing Performance: Triggers any remaining JIT compilations and establishes baseline performance. The Key-Value (KV) Cache for attention mechanisms is also initialized, though it's typically request-specific.

In serverless or containerized deployments (e.g., AWS Lambda, Kubernetes), cold start includes the time to launch the runtime environment itself:

• Container Image Pull: Downloading and unpacking the container layers from a registry.
• Runtime Init: Starting the language runtime (Python, Node.js), importing libraries (PyTorch, Transformers), and loading the serving application code.
• This component is often the largest variable, ranging from hundreds of milliseconds to several seconds.

In managed serving platforms, additional orchestration steps add to the observed latency:

• Scheduler Decision Time: The cluster scheduler (e.g., Kubernetes) finding a suitable node and binding the pod.
• Health Check Pass-Through: The service must pass initial liveness and readiness probes before being added to the load balancer pool. Configured check intervals directly add to the first-request delay.
• Service Mesh Sidecar Injection: If used, sidecar proxies (e.g., Envoy) must also initialize and establish connections.

Each component has targeted mitigation strategies:

• Model Loading: Use provisioned concurrency (serverless) or replica pre-pulling (Kubernetes) to keep instances warm.
• Hardware Init: Employ persistent GPU contexts and AOT-compiled engines (TensorRT) to eliminate JIT overhead.
• Warm-Up: Execute a scripted series of dummy inferences post-load.
• Container Startup: Optimize image size, use cached or pre-warmed images, and leverage snapshotting (Firecracker).
• Baseline Measurement: A typical cold start for a large language model can range from 2 seconds (optimized container) to >30 seconds (full initialization from scratch).

The most direct mitigation is to preload models into memory before they receive live traffic. This is achieved through warm-up scripts that send synthetic requests to the serving system at startup or during low-traffic periods. Orchestrators like Kubernetes can use init containers or startup probes to hold traffic until the model is fully loaded and its KV cache is initialized. The goal is to transition the model from a cold disk state to a hot, in-memory state before any user request arrives.

Instead of scaling to zero, predictive autoscaling maintains a minimum number of warm instances based on traffic forecasts (e.g., daily cycles). Keep-alive policies prevent idle instances from terminating prematurely during short lulls. For serverless platforms, provisioned concurrency reserves a set of pre-initialized execution environments. This technique trades a small, constant resource cost for the elimination of unpredictable cold start penalties during traffic surges.

Reducing the model's on-disk footprint directly cuts loading time. Key methods include:

• Quantization: Converting weights from FP32 to INT8 or FP16.
• Pruning: Removing non-essential neurons or weights.
• Compiler Optimization: Using frameworks like TensorRT or OpenVINO to create a fused, platform-specific model execution graph (e.g., a .plan or .onnx file).
• Efficient Serialization: Formats like Safetensors or protocol buffers load faster than standard PyTorch checkpoints. These optimizations shrink the artifact that must be read from disk and deserialized.

Implementing a multi-tier caching strategy isolates the cold start to the slowest, least-frequent layer.

• In-Memory Cache (Hot): The model's weights and KV cache reside in GPU/CPU RAM.
• Local SSD Cache (Warm): A fast disk cache on the compute instance (e.g., NVMe).
• Remote Object Store (Cold): The source of truth (e.g., S3, GCS). On a cold start, the system first checks the local SSD before fetching from the remote store, which can be orders of magnitude faster. Peer-to-peer caching between instances in a cluster can further accelerate population.

This advanced technique anticipates which model will be needed next. Based on routing logic (user session, API endpoint), the system can speculatively begin fetching and loading a model while the current request is being processed. Just-in-time fetching overlaps the network I/O of downloading the model artifact with other initialization steps. This requires sophisticated scheduling but can make the model ready precisely when needed, effectively hiding the load time from the user.

When cold starts are unavoidable, the user experience can be preserved by architectural decoupling. Instead of a synchronous request-response, the system immediately acknowledges the request with a job ID and queues the work. The client polls for completion or receives a webhook callback. This changes the performance SLO from time to first token (TTFT) to time to job acceptance, which can be sub-10ms even during a cold start. This pattern is common in batch processing and some agentic workflows.

The total time delay between submitting an input to a machine learning model and receiving its corresponding output. This is the overarching category that includes cold start latency, network transmission, compute processing, and queuing delays. It is the primary user-facing performance metric for any AI service.

The duration from the start of an inference request to when the first token of the output is generated or delivered. This is heavily impacted by cold starts, as the model loading and initial prefilling phase must complete before any tokens can be streamed. A high TTFT directly harms perceived responsiveness in chat and streaming applications.

The time required for a language model to process the static input prompt through its initial forward pass, generating the Key-Value (KV) cache before autoregressive token generation begins. This phase is a major contributor to cold start latency and TTFT, as the entire context must be processed sequentially.

• Impact: Scales with prompt length.
• Optimization: Techniques like continuous batching and attention optimization (e.g., FlashAttention) target this phase.

An inference optimization technique where new requests are dynamically added to a running batch as previous requests finish generation. This maximizes GPU utilization and throughput, which indirectly reduces cold start impact by:

• Keeping Models Warm: A continuously batched server keeps models loaded, serving many requests between cold starts.
• Reducing Queuing Delay: Higher throughput means requests spend less time waiting, improving overall end-to-end latency.

A target reliability goal defined for a specific latency percentile (e.g., P99 < 500ms). Cold start events are a primary cause of SLO violations in low-traffic or autoscaling scenarios. Defining and monitoring SLOs requires:

• Distinguishing Cold vs. Warm Requests: Segmenting latency metrics to understand the true baseline and cold start penalty.
• Error Budget Management: Accounting for the latency spikes caused by cold starts in deployment and scaling policies.