Payload Size

The number of tokens in the input prompt is the most direct contributor to payload size. In transformer-based models, this directly scales the computational cost of the prefill phase, where the model processes the entire context to build the initial Key-Value (KV) cache.

• Direct Scaling: Latency for the prefill phase typically increases linearly or sub-linearly with input token count.
• Context Window Limits: Models have fixed maximum context windows (e.g., 128K tokens). Exceeding this requires truncation or specialized processing.
• Example: A 10,000-token document summarization request has a fundamentally larger computational payload than a 50-token chat message.

The requested or generated number of output tokens defines the size of the response payload and dictates the duration of the autoregressive decoding phase. Each generated token becomes part of the response and requires a forward pass.

• Decoding Latency: Time Per Output Token (TPOT) is multiplied by the total output length.
• Streaming vs. Non-Streaming: In streaming responses, Time to First Token (TTFT) is critical, but total completion time is governed by output length and TPOT.
• Stopping Criteria: Generation stops based on max_tokens parameters, stop sequences, or end-of-sequence tokens, directly controlling final payload size.

Before transmission, structured request/response data (prompt, parameters, generated text) must be converted to/from a byte stream. This process adds CPU-bound latency.

• Common Formats: Protocol Buffers (gRPC), JSON, and MessagePack are standard. Protobuf is typically more efficient than JSON.
• Cost Factors: Overhead scales with payload size and complexity (e.g., nested tool-calling specifications).
• Measurement: This overhead is included in gRPC latency and contributes to the difference between model execution time and end-to-end latency.

The time required to transmit the serialized payload bytes over the network. This is governed by physical laws (speed of light) and network conditions (bandwidth, congestion).

• Formula: Transfer Time ≈ Payload Size (bits) / Available Bandwidth (bits/sec).
• Impact on E2E Latency: For large payloads (e.g., multi-page document inputs, long completions) on limited bandwidth links, this can dominate total latency.
• Mitigation: Compression (e.g., gzip) reduces transfer size at the cost of added CPU time for compression/decompression.

During inference, transformers store intermediate Key and Value states for each token in context to avoid recomputation. This KV cache is a major in-memory payload.

• Memory Scaling: Cache size scales as (batch_size * sequence_length * num_layers * num_heads * head_dim * 2 * dtype_size).
• Optimization Techniques: PagedAttention (vLLM) and quantized caching (FP8/INT8 KV cache) are used to manage this footprint.
• Bottleneck: Under high concurrency, KV cache memory can exhaust GPU VRAM, leading to out-of-memory errors or forced recomputation, severely impacting latency.

Payloads containing images, audio, or other non-text data are significantly larger. These inputs require preprocessing and encoding into the model's embedding space.

• Raw Data Size: A single high-resolution image can be megabytes, compared to kilobytes for equivalent text.
• Encoding Latency: Specialized vision encoders (e.g., CLIP, SigLIP) must process the raw pixels, adding a substantial, payload-size-dependent preprocessing step before the main model inference.
• Example: A request asking a vision-language model to analyze ten product images will have a payload orders of magnitude larger than a text-only query.

The time a language model spends processing the static input prompt through its initial forward pass to generate the first Key-Value (KV) cache. This phase is highly sensitive to prompt length (a major component of input payload size).

• A long context window (e.g., 128K tokens) can make prefill the dominant latency cost for short outputs.
• Optimizations include attention slicing and flash attention to accelerate this computationally intensive step.

The fundamental engineering trade-off where increasing system throughput (Queries Per Second) typically increases latency, especially tail latency (P99). Payload size exacerbates this.

• Continuous batching improves throughput but can increase latency for individual requests if the batch is not full.
• Larger payloads reduce the effective batch size that fits in GPU memory, lowering maximum throughput.
• The throughput-latency curve is used to find the optimal operating point for a given payload profile.