Cold Start

The most significant contributor to cold start is loading the model weights from persistent storage (e.g., network-attached storage, object store) into GPU memory. For a multi-billion parameter model, this involves transferring tens to hundreds of gigabytes. The delay is a function of network bandwidth and storage I/O speed. For example, loading a 70B parameter model in FP16 (~140GB) over a 10 Gbps network incurs a minimum of ~112 seconds of pure data transfer time, not accounting for deserialization overhead.

In serverless or containerized environments (e.g., AWS Lambda, Kubernetes), a cold start requires spinning up a new runtime environment. This involves:

• Pulling the container image from a registry, which can be several gigabytes in size for a full ML stack.
• Installing system and Python dependencies.
• Loading the inference framework (e.g., PyTorch, TensorRT-LLM). These steps add substantial overhead before the model loading even begins, especially if images are not pre-cached on the host.

For peak performance, frameworks often compile the model's computational graph for the specific hardware. This just-in-time (JIT) compilation is a major cold start factor for optimized runtimes like:

• TensorRT-LLM: Builds and optimizes an engine for the target GPU.
• ONNX Runtime: Performs graph optimizations and kernel selection.
• XLA (used with JAX/PyTorch): Compiles the model for efficient execution. This compilation can take from seconds to minutes, depending on model size and optimization level, but is cached for subsequent warm requests.

Large models that exceed the memory of a single GPU must be split using model parallelism techniques like tensor parallelism or pipeline parallelism. Cold start for these models includes:

• Partitioning and loading weights across multiple GPUs/nodes.
• Establishing high-speed communication channels (e.g., NCCL) between devices.
• Synchronizing all processes before serving can begin. The coordination overhead scales with the number of devices and the complexity of the parallelization strategy, adding significant latency.

Before the endpoint is ready for production traffic, internal caches and data structures must be initialized. This includes:

• Populating the KV Cache data structures in memory, setting up the paging mechanism (e.g., vLLM's PagedAttention).
• Running warm-up inference passes with dummy data to trigger CUDA kernel compilation and memory allocation, ensuring the first real request doesn't pay this cost.
• Loading tokenizers, vocabulary files, and any auxiliary models (e.g., for embedding generation in RAG setups).

In managed serving platforms (e.g., Kubernetes with Kserve, SageMaker), the cold start latency also includes the control plane overhead:

• Orchestrator decision time: The time for the Horizontal Pod Autoscaler (HPA) or cluster autoscaler to detect need and schedule a new pod.
• Node provisioning: If no worker nodes with sufficient resources are available, the cloud provider must provision a new virtual machine, which can take minutes.
• Health check passing: The new pod must start, pass readiness probes, and be added to the service mesh's load balancing pool before receiving traffic.

A cloud computing execution model where the provider dynamically manages server allocation. Cold starts are a defining characteristic, occurring when a new instance must be spun up to handle a request after a period of inactivity. This model abstracts infrastructure management but introduces variable latency.

• Key Trade-off: Developer agility vs. predictable performance.
• Example: AWS Lambda, Google Cloud Functions.

A hosted API that exposes a machine learning model for predictions. Managing cold start for these endpoints is crucial, as it involves loading a potentially multi-gigabyte model into GPU memory. Strategies include pre-warming (sending dummy requests) and maintaining a minimum number of active instances.

• Primary Challenge: Balancing cost (keeping instances warm) with latency (handling sudden traffic spikes).

A Kubernetes resource that automatically scales the number of pod replicas based on observed metrics like CPU or memory. It interacts directly with cold start dynamics:

• Scale-up: Adding a new pod triggers a cold start as the container initializes and loads the model.
• Scale-to-zero: Setting replicas to zero saves cost but guarantees a cold start on the next request. Effective scaling policies must account for model load time to meet latency SLOs.

The overarching process of deploying a trained model into production to handle inference requests. Cold start is a key operational hurdle within this process. Specialized serving engines implement optimizations to mitigate it:

• vLLM: Uses PagedAttention to manage the KV Cache efficiently, allowing faster loading and execution.
• Text Generation Inference (TGI): Implements continuous batching to improve throughput once warm, making the cold start penalty a smaller fraction of total workload.

A target level of reliability or performance for a service, such as a 99.9% availability or a p95 latency of 200ms. Cold start latency is a major threat to latency SLOs, especially for serverless functions or infrequently used model endpoints. Engineering efforts to mitigate cold starts (e.g., provisioned concurrency, keep-warm pings) are directly driven by these contractual performance targets.

A model compression technique that reduces the numerical precision of weights and activations (e.g., from FP16 to INT8). This directly attacks a root cause of cold start time by shrinking the model size that must be loaded into memory.

• Impact: Smaller models load faster from disk or network storage into GPU RAM.
• Method: Post-Training Quantization (PTQ) is commonly used to quantize LLMs without retraining, significantly reducing both memory footprint and initial load latency.