Cold Start

The primary source of latency is the time required to provision and boot a new compute instance (virtual machine or container) from a stopped or non-existent state. This process involves:

• Image Pulling: Downloading the container image or VM snapshot containing the model binaries, dependencies, and serving runtime.
• Resource Allocation: The cloud provider's scheduler must locate and assign physical hardware (e.g., a specific GPU).
• OS and Runtime Boot: The guest operating system and application runtime (e.g., Python, CUDA drivers) must initialize before the model can load. This initialization can take from several seconds to over a minute, depending on image size and provider.

After the instance is ready, the model's parameters must be transferred from persistent storage (e.g., network-attached disk) into the GPU's high-bandwidth memory (VRAM). This is a major bottleneck.

• I/O Bound Transfer: Loading a multi-gigabyte model file over a network is slow compared to in-memory computation.
• VRAM Allocation: The GPU driver must allocate contiguous memory blocks for the model weights, which can be a complex operation for large models.
• Weight Initialization: Some frameworks perform additional setup, converting weights to the optimal layout for the specific GPU architecture. For a 70B parameter model in FP16, this can involve moving over 140 GB of data.

Modern cloud-native deployments often use horizontal pod autoscalers (Kubernetes) or similar mechanisms that scale the number of serving replicas to zero during periods of no traffic to save costs. When a new request arrives, the orchestrator must:

1. Detect the need for a new replica.
2. Schedule it on an available node (which may itself need scaling).
3. Wait for the full cold start sequence to complete. While cost-effective, this policy directly trades off savings for guaranteed latency spikes on the first request after an idle period.

The abstraction layers that enable portability and scalability introduce inherent latency. Key contributors include:

• Container Runtime Startup: Engines like containerd or docker must create namespaces, cgroups, and mount filesystems.
• Sidecar Containers: In service meshes (e.g., Istio), a request cannot be routed until the proxy sidecar container is fully initialized.
• Readiness Probes: The orchestrator will not send traffic to a pod until its health checks pass, which typically requires the model server to be fully loaded and listening. This adds verification time to the critical path.

The performance gap that defines the cold start problem. Key differences:

Cold starts are a primary driver of poor tail latency metrics (P95, P99). While most requests hit warm instances and are fast, a small percentage that trigger new instance creation experience orders-of-magnitude higher latency.

• Sporadic Traffic Patterns: Applications with bursty or unpredictable traffic suffer most, as instances scale down during lulls.
• User Experience Degradation: For interactive applications like chatbots, a 20-second cold start is catastrophic for user retention. Mitigation strategies like predictive scaling or maintaining a warm pool of instances directly target improving these worst-case latency percentiles.

A cloud resource management strategy that automatically adjusts the number of active compute instances in a serving cluster based on real-time metrics like request queue length or GPU utilization. It directly combats cold starts by maintaining a warm pool of instances during predicted demand, but must be tuned to balance provisioning latency against idle resource cost.

• Predictive Scaling: Uses historical traffic patterns to pre-warm instances before demand spikes.
• Reactive Scaling: Adds instances after a metric threshold (e.g., CPU > 70%) is crossed, incurring a cold start penalty.
• Key Trade-off: Over-provisioning wastes money on idle instances; under-provisioning causes user-facing cold start delays.

vLLM is a high-throughput inference serving engine that uses the PagedAttention algorithm to optimize KV cache memory management. This reduces the memory footprint of each concurrent request, allowing more model instances to be kept resident in GPU memory on a warm server.

• Mechanism: PagedAttention manages the KV cache in non-contiguous blocks, drastically reducing memory fragmentation.
• Impact on Cold Start: By improving memory efficiency, vLLM enables higher request concurrency per GPU. This increases the likelihood of a warm instance being available for an incoming request, reducing the frequency of cold starts.

Refers to the worst-case response times experienced by a small fraction of user requests, measured at high percentiles like the 95th (P95) or 99th (P99). Cold starts are a primary driver of poor tail latency in LLM services.

• User Impact: While average latency may be good, a high P99 latency means 1% of users suffer very slow responses, often due to hitting a cold instance.
• Monitoring: Engineering teams track P95/P99 latency alongside average latency to ensure consistent quality of service and identify cold start problems.

The total financial expenditure of running a trained model to make predictions. Cold starts directly increase inference cost through inefficient resource utilization.

• Cost Components: GPU/CPU time, memory, networking, and serving infrastructure.
• Cold Start Penalty: Paying for GPU time while the model loads, but generating zero tokens for the user. This wasted compute inflates the effective cost per token.
• Optimization Goal: Minimize the ratio of idle/warm-up time to productive inference time.

A fault tolerance mechanism where a system under extreme load proactively rejects or delays low-priority requests to prevent catastrophic failure. It interacts with cold start management.

• Use Case: During a traffic surge, if the autoscaler cannot provision warm instances fast enough, the system may shed low-priority traffic rather than subject all users to cold start delays.
• Priority Queues: Requests can be classified (e.g., interactive vs. batch). Shedding ensures high-priority requests get the warm instances, while others wait or are rejected.

A compression technique that reduces the numerical precision of a model's weights and activations (e.g., from FP16 to INT8). This reduces the model size and the time required to load it into GPU memory, directly shortening the cold start period.

• INT8/FP8 Quantization: Common precision targets that reduce memory footprint by 50-75% vs. FP16.
• Faster Loading: Smaller models load from disk or network into VRAM more quickly.
• Trade-off: Aggressive quantization (e.g., to 4-bit) can impact model quality, requiring careful evaluation.