Cold Start

The primary source of cold start latency is the time required to read model weights from persistent storage into GPU or CPU memory. For large language models, this can involve transferring hundreds of gigabytes over network or disk interfaces.

• Model Size Impact: A 70B parameter model in FP16 requires ~140GB of memory, taking minutes to load even on high-bandwidth NVMe storage
• Disk I/O Constraints: Traditional block storage may only sustain 1-2 GB/s read throughput, creating a hard floor on initialization time
• Network-Attached Storage: Loading from cloud object stores like S3 adds network latency and throughput variability, often making this the slowest path
• Memory Mapping: Techniques like mmap can defer page faults to first access, reducing apparent startup time but shifting latency to the first real inference request

In Kubernetes-orchestrated environments, cold start latency includes the full container lifecycle: image pulling, pod scheduling, and resource allocation before the model process even begins loading weights.

• Image Pull Time: Large container images containing CUDA runtimes and framework dependencies can take 30-60 seconds to pull on a cold node
• Node Autoscaling Delay: If no GPU node is available, the cluster autoscaler must provision a new VM, adding 2-5 minutes before pod scheduling begins
• Init Container Execution: Pre-loading steps like downloading model artifacts or setting up shared memory segments add sequential delays
• Health Check Gating: Readiness probes must pass before traffic is routed, meaning the model must be fully loaded and responsive before serving begins

When a model runs on a GPU for the first time, the CUDA or TensorRT runtime must just-in-time compile optimized kernels for the specific operations and tensor shapes in the model graph. This one-time compilation adds significant latency to the first inference.

• CUDA JIT Compilation: PyTorch's CUDA graphs and custom kernels are compiled on first execution, adding seconds to the initial forward pass
• TensorRT Build Phase: Converting an ONNX model to a TensorRT engine requires layer fusion, kernel selection, and precision calibration, which can take minutes
• Shape Polymorphism: Dynamic input shapes prevent full ahead-of-time compilation, forcing recompilation when new batch sizes or sequence lengths are encountered
• Cache Persistence: Compiled kernels can be serialized to disk and reloaded on subsequent starts, eliminating this overhead for repeat deployments

For generative transformer models, the key-value cache must be allocated in GPU memory before autoregressive decoding can begin. This allocation and zeroing of large memory buffers contributes to cold start latency.

• Memory Allocation: Allocating contiguous GPU memory for the KV cache across all transformer layers requires significant time for large context windows
• PagedAttention Pre-allocation: Systems like vLLM pre-allocate KV cache blocks to enable efficient memory management, but this upfront allocation extends startup time
• Context Length Scaling: A model configured for 128K token contexts requires proportionally larger KV cache buffers, increasing initialization time linearly
• NUMA-Aware Placement: Ensuring KV cache memory is allocated on the NUMA node local to the GPU minimizes access latency but requires careful orchestration during startup

Production systems employ several techniques to eliminate or hide cold start latency from end users, ensuring that autoscaling events do not degrade the customer experience.

• Pre-Warming: Sending synthetic 'dummy' inference requests to newly started instances forces weight loading, kernel compilation, and cache allocation before real traffic arrives
• Over-Provisioning: Maintaining a buffer of idle, fully-loaded instances ready to absorb traffic spikes eliminates cold starts at the cost of higher baseline infrastructure spend
• Lazy Loading with Fallback: Routing initial requests to a fast but less accurate fallback model while the primary model loads, then seamlessly switching over
• Snapshot Restoration: Serializing the fully-initialized model state (weights + compiled kernels + KV cache) to disk and restoring it via direct memory copy, reducing startup from minutes to seconds

Cold start latency directly determines how quickly a model serving system can react to traffic surges. The total scale-out latency is the sum of detection time, infrastructure provisioning, and model initialization.

• Scale-Out Lag: A 3-minute cold start means the system is under-provisioned for 3 minutes during a traffic spike, causing request queuing and P99 latency spikes
• HPA Synchronization: The Horizontal Pod Autoscaler's default 15-second metric polling window adds detection delay on top of cold start time
• Thrashing Risk: If cold start time exceeds the scale-down cooldown period, instances may be terminated before they finish initializing, creating a destructive cycle
• Predictive Scaling: Time-series forecasting of traffic patterns can trigger pre-emptive scaling before demand materializes, hiding cold start latency entirely from users

The total time elapsed between a client sending an inference request and receiving the complete prediction response. Cold start contributes directly to tail latency, as the first request to a newly initialized instance includes the overhead of weight loading and memory allocation. This metric is often decomposed into:

• Queue time: waiting for a worker
• Initialization time: loading model artifacts
• Compute time: the actual forward pass

A percentile metric indicating that 99% of inference requests are served faster than this value. Cold start events disproportionately inflate P99 and P999 latencies because the unlucky requests that hit a cold instance experience orders-of-magnitude slower responses. Engineering teams target P99 thresholds in their Service Level Objectives (SLOs) precisely to catch these initialization spikes.

A model optimization technique that reduces the numerical precision of weights and activations from 32-bit floats to lower-bit integers like INT8 or INT4. Quantization directly mitigates cold start by:

• Reducing the total bytes that must be read from disk into memory
• Decreasing the time required for memory allocation on the accelerator
• Enabling faster deserialization of model artifacts Smaller models load faster, shrinking the initialization window.