Prompt Throughput

The total number of tokens in the input prompt and conversation history is the primary determinant of computational load. Longer contexts require more sequential attention computations and increase memory bandwidth pressure for the Key-Value (KV) Cache.

• Impact: Throughput often scales inversely with prompt length. Doubling the context can more than halve the tokens/sec rate.
• Optimization: Techniques like prompt caching for static prefixes or sliding window attention can mitigate this cost for long contexts.

The base model's parameter count and architectural choices directly constrain throughput. Larger models have higher latency per token due to increased compute per layer and memory footprint.

• Attention Mechanism: Models using FlashAttention or other optimized kernels achieve significantly higher throughput by reducing memory I/O.
• Parameter Efficiency: PEFT methods like Prompt Tuning or LoRA keep the base model frozen, avoiding the throughput penalty of full fine-tuning while adding minimal overhead for the small adapter weights.

Deployment-level optimizations are essential for maximizing hardware utilization and throughput.

• Continuous Batching: Dynamically batches incoming requests of varying lengths, dramatically improving GPU utilization compared to static batching.
• Quantization: Using FP16, INT8, or NF4 precision reduces model memory footprint and increases compute speed, often with negligible accuracy loss.
• Speculative Decoding: Uses a smaller 'draft' model to propose tokens, which are then verified in parallel by the larger target model, increasing decoding speed.

The underlying compute infrastructure sets the physical ceiling for throughput.

• GPU Memory Bandwidth: Often the bottleneck for large model inference, especially during the prefill phase (processing the prompt).
• Kernel Fusion: Fusing multiple operations (e.g., layer normalization, activation) into a single GPU kernel reduces launch overhead and data movement.
• Tensor Parallelism: Splits model layers across multiple GPUs to handle larger models or increase batch sizes.

The nature of the task influences both the prefill (prompt processing) and decode (generation) phases.

• Reasoning Tasks: Require longer internal chains of thought, increasing compute time per output token.
• Long-Form Generation: Tasks requiring hundreds of output tokens shift the bottleneck from prompt processing to the autoregressive decode phase.
• Tool Calling/Retrieval: Systems that pause generation to call external APIs or search a vector database have lower effective token throughput due to added latency, even if model inference is fast.

The efficiency of the serving software and runtime libraries has a major impact.

• Serving Framework: Specialized systems like vLLM, TGI (Text Generation Inference), or TensorRT-LLM provide optimized kernels, efficient scheduling, and paged attention for superior throughput.
• KV Cache Management: Efficient management of the KV Cache in memory is critical. PagedAttention (used in vLLM) eliminates fragmentation and allows near-optimal memory use for variable-length sequences.
• Communication Overhead: In multi-GPU or distributed settings, latency from cross-device communication can become a bottleneck.

In prefix tuning, the trainable prefix can be generated by a small prompt encoder (e.g., an MLP). An optimization involves pre-computing and caching the output of this encoder, or fusing its operations, to avoid executing this extra network during each inference step.

• Advanced Technique: For fixed tasks, the prompt encoder can be compiled away, and its output—the final prefix vectors—can be stored directly as the cached prompt.
• Benefit: Removes the small but cumulative overhead of running the encoder network, streamlining the inference path to be as simple as standard prompt caching.

Used in multi-task or continual learning scenarios with a prompt pool. Instead of applying all learned prompts simultaneously, a lightweight classifier or retrieval mechanism selects the most relevant single prompt (or sparse combination) for a given input query.

• How it works: A small model (e.g., a distilled BERT) analyzes the input and retrieves the corresponding prompt embedding from the pool before the main LLM forward pass.
• Throughput Gain: Prevents the computational cost from scaling linearly with the number of prompts in the pool. Only the parameters for the selected prompt are activated, maintaining inference cost close to that of a single prompt.

Designing the prompt length and structure with the underlying model's architecture in mind. For example, aligning the length of a soft prompt to match the model's attention window or block size can prevent inefficient padding and memory access patterns.

• Technical Consideration: Using a prompt length that is a multiple of the GPU's optimal tensor core size (e.g., 64 or 128) can improve hardware utilization.
• Trade-off: This involves finding the shortest prompt length that achieves target task performance, as every additional virtual token adds linear computational cost during the prefill phase.

The time delay introduced during inference by processing the prompt. This includes the computational overhead of applying a tuned soft prompt or prefix and is a critical factor for real-time applications. High latency can bottleneck prompt throughput. Factors influencing it include:

• Prompt length: Longer continuous prompts require more forward passes.
• Model size: Larger base models have slower per-token processing.
• Hardware: Inference is constrained by GPU memory bandwidth and compute.
• Caching: Techniques like prompt caching for static prefixes can reduce latency.

An inference optimization technique where the computed key-value states (KV cache) for a static prefix or system prompt are stored and reused across multiple generation requests. This avoids redundant computation for the fixed prompt portion, directly improving prompt throughput and reducing prompt latency. It is highly effective for:

• Batch inference: Processing multiple user queries with a shared system prompt.
• Multi-turn dialogues: Reusing cached context from previous turns.
• Serverless deployments: Where cold starts benefit from pre-computed states.

The number of virtual tokens or continuous embeddings used in a soft prompt or prefix. This is a critical hyperparameter that balances expressiveness and computational efficiency.

• Longer prompts can capture more complex task instructions but increase memory usage and computation, reducing throughput.
• Shorter prompts are more efficient but may lack the capacity to fully steer the model. Optimal length is task-dependent and must be tuned to maximize performance per unit of computational cost.

A broad pillar covering techniques to reduce compute costs and latency during model execution. Methods directly impacting prompt throughput include:

• Continuous batching: Dynamically batches incoming requests of varying lengths to maximize GPU utilization.
• Quantization: Reduces the numerical precision of model weights (e.g., to 8-bit or 4-bit) to decrease memory footprint and increase speed.
• Flash Attention: Optimizes the attention mechanism to be more memory-efficient and faster.
• Speculative decoding: Uses a smaller draft model to propose tokens, verified by the main model, to accelerate generation.

A discrete sequence of human-readable tokens or natural language instructions used to guide a model's behavior. Contrasted with a learned soft prompt.

• Impact on Throughput: Hard prompts are processed identically to user input tokens. Their length adds directly to the sequence length, affecting total generation time.
• Engineering vs. Tuning: Prompt engineering crafts hard prompts manually, while prompt tuning learns soft prompts via gradients.
• Efficiency: Hard prompts require no training but may need extensive iterative testing, whereas soft prompts incur training cost but can be more compact and performant.

A more granular performance metric than request throughput, measuring the number of tokens a system can process per second. It is the fundamental unit behind prompt throughput.

• Calculation: (Prompt Tokens + Generated Tokens) / Total Generation Time.
• Bottlenecks: Affected by memory bandwidth (loading model weights), compute FLOPs (matrix multiplications), and autoregressive dependency (generating tokens one-by-one).
• Optimization Goal: Maximize tokens/second/dollar, which involves trade-offs between model size, batch size, and sequence length.