Compute Optimization

A compression technique that reduces the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). This decreases memory footprint and accelerates computation on hardware with optimized integer units, with minimal accuracy loss.

• INT8 Quantization: Reduces memory by 75% vs. FP32.
• GPTQ & AWQ: Advanced algorithms for aggressive 4-bit quantization.
• Primary Benefit: Enables larger models to run on less expensive or more available hardware.

Optimizes the memory and compute of the transformer's attention mechanism, the primary bottleneck during generation.

• KV Cache: Stores computed key/value tensors for previous tokens, eliminating redundant computation for each new token. This is the single most impactful latency reduction for autoregressive decoding.
• PagedAttention: An algorithm (used by vLLM) that manages the KV cache in non-contiguous blocks, drastically reducing memory waste and fragmentation, allowing higher concurrency.
• FlashAttention: A seminal IO-aware algorithm that recomputes attention on-chip to avoid reading/writing large intermediate matrices to slow GPU memory, speeding up training and inference.

Maximizes GPU utilization by grouping multiple user requests for parallel processing.

• Continuous Batching (In-flight Batching): Dynamically adds new requests to a running batch as previous requests finish generation, eliminating idle GPU time. This is superior to static batching for variable-length conversations.
• Dynamic Batching: Groups requests into a batch based on current load, adjusting batch size in real-time to balance latency and throughput.
• Impact: Can improve throughput by 5-10x compared to processing requests individually, directly lowering cost per token.

An inference acceleration technique that uses a smaller, faster model to 'draft' multiple future tokens, which are then verified in parallel by the larger target model.

• Mechanism: The draft model proposes a candidate sequence (e.g., 3-5 tokens). The main model runs these tokens through a single, parallel forward pass, accepting all correct tokens and regenerating only from the first incorrect one.
• Benefit: Can achieve 2-3x latency reduction for the same model, as the larger model's forward passes are used more efficiently.
• Use Case: Ideal for applications where a small, aligned draft model is available.

Techniques to create smaller, faster models from larger ones.

• Model Pruning: Removes redundant or less important weights/neurons from a trained network. Methods include magnitude pruning (removing smallest weights) and structured pruning (removing entire channels).
• Knowledge Distillation: Trains a compact 'student' model to mimic the outputs and internal representations of a large 'teacher' model. The student achieves comparable performance with far fewer parameters.
• Result: Creates models that are cheaper to serve and can be deployed on edge or less powerful hardware.

Infrastructure and traffic management techniques that optimize aggregate resource use across a serving cluster.

• Autoscaling: Dynamically adds or removes GPU instances based on request load (queue length, token throughput), matching capacity to demand to minimize idle cost.
• Rate Limiting & Load Shedding: Protects the system from being overwhelmed. Rate limiting controls request flow per user/key, while load shedding proactively rejects low-priority requests during overload to maintain latency for critical traffic.
• Instance Right-Sizing: Selecting the most cost-effective cloud GPU instance (e.g., A10G vs. H100) based on a model's specific memory and compute needs.

Tokens Per Second (TPS) is the primary throughput metric for LLM inference, measuring the raw output generation speed of a system. It directly impacts user experience and total cost.

• Key Drivers: Batch size, model architecture, GPU compute capability, and memory bandwidth.
• Trade-off with Latency: Maximizing TPS often involves larger batch sizes, which can increase per-request latency.
• Optimization Target: The goal is to achieve the highest sustainable TPS for a given latency Service Level Agreement (SLA) and hardware budget.

Latency measures the delay a user experiences. Tail latency (P95/P99) captures the worst-case delays for the slowest 5% or 1% of requests, critical for consistency.

• Time to First Token (TTFT): The delay from request send to receiving the first output token. Dominated by prefill computation and model loading (cold start).
• Time Per Output Token (TPOT): The incremental delay for each subsequent token, driven by autoregressive generation speed.
• Governance: SLAs are typically defined against P99 latency to guarantee quality of service for all users.

Cost Per Token is the fundamental unit economics metric, calculating the amortized expense of generating a single token. It is the ultimate measure of inference efficiency.

• Calculation: (Instance Cost per Hour) / (Tokens Generated per Hour).
• Primary Levers: Model quantization, dynamic batching, high GPU utilization, and selecting cost-optimal instance types.
• GPU Utilization: A leading indicator of hardware efficiency. Low utilization signals wasted capacity, often addressed via continuous batching and autoscaling.

FinOps is the operational practice of managing cloud spend. For LLMs, it involves collaborative governance between engineering and finance teams.

• Cloud Cost Allocation: Attributing GPU and infrastructure costs to specific teams, projects, or API keys using resource tagging. Enables accurate chargeback/showback.
• Instance Right-Sizing: Continuously analyzing workload patterns to select the most cost-effective GPU instance (e.g., A10G vs. H100) without over-provisioning.
• Budget Alerts & Anomaly Detection: Automated monitoring to flag unexpected spend spikes from model changes or traffic surges.

Concurrency measures the number of simultaneous requests a system can handle while meeting latency SLAs. It is a function of throughput and latency.

• Autoscaling Triggers: Scale decisions are based on metrics like request queue length, GPU utilization, or concurrency itself.
• Cold Start Impact: Scaling out adds new instances, incurring cold start latency as models load. This must be factored into scaling policies.
• Load Shedding & Rate Limiting: Under extreme load, these mechanisms protect system stability by rejecting low-priority requests (load shedding) or limiting user quotas (rate limiting).

Memory is a critical bottleneck. Key metrics track the efficiency of memory usage for model weights and the KV Cache.

• Model Memory Footprint: Size of loaded model parameters, reduced via quantization (e.g., INT8, GPTQ) and pruning.
• KV Cache Memory per Request: Directly proportional to sequence length and batch size. Managed by algorithms like PagedAttention in vLLM to reduce fragmentation.
• Memory-Bound vs. Compute-Bound: Identifying the limiting factor (memory bandwidth vs. FLOPs) guides optimization strategy (e.g., quantization vs. kernel optimization).

The financial analysis of running LLMs in production. Key metrics for CTOs and FinOps teams include:

• Cost Per Token: The foundational unit economics metric for forecasting and budgeting.
• Tokens Per Second (TPS): A throughput metric directly tied to infrastructure efficiency and user experience.
• Tail Latency (P95/P99): The worst-case response times critical for service-level agreements (SLAs). Optimization aims to minimize cost per token while maintaining required TPS and latency bounds, often involving trade-offs between these metrics.