Cost Per Token

The foundational driver of cost. Larger models with more parameters (e.g., 70B vs. 7B) require more memory and compute per token.

• Key Factor: The number of parameters directly correlates with the FLOPs (floating-point operations) required for a single forward pass.
• Example: Generating a token with a 70B parameter model is inherently more computationally expensive than with a 7B model, leading to a higher base cost per token.

The numerical format of model weights dramatically impacts memory bandwidth and compute efficiency.

• FP16/BF16: Higher precision, standard for training and high-accuracy inference.
• INT8/FP8: Quantized formats that reduce memory footprint and can accelerate compute on supported hardware, lowering cost per token.
• GPTQ/AWQ (4-bit): Advanced post-training quantization methods that enable running very large models on consumer GPUs, offering the lowest cost per token but with potential accuracy trade-offs.

The unit economics of the underlying cloud or on-premises infrastructure.

• GPU Type: An NVIDIA H100 provides vastly higher throughput (tokens/sec) than an A10G, affecting the cost allocated to each token.
• Memory Bandwidth: A critical bottleneck for inference; higher bandwidth (e.g., HBM3) allows for faster data loading, improving tokens/sec.
• Cloud Pricing: The hourly rate of the chosen instance (e.g., g5.48xlarge) is divided by its throughput to determine a baseline hardware cost per token.

Serving techniques that maximize hardware utilization, directly reducing the effective cost per token.

• Continuous Batching: Groups multiple user requests (sequences) into a single batch, dramatically improving GPU utilization and throughput.
• PagedAttention (vLLM): Eliminates memory waste in the KV Cache, allowing for higher concurrency and more requests served per GPU.
• Speculative Decoding: Uses a small draft model to propose tokens, verified in parallel by the larger model, increasing overall tokens/sec.

The nature of the input and output directly influences computational work.

• Context Length: Longer input prompts require more initial computation to process the full context, amortized over the generated tokens.
• Output Length: Generating a 1000-token response incurs roughly 1000x the cost of a 1-token response, as each new token requires an autoregressive step.
• Cache Hit Rate: For repeated or similar prompts, a warm KV Cache can skip recomputation, lowering the cost for subsequent tokens.

The real-world costs of running a production service beyond raw computation.

• Load Patterns: Variable traffic leads to idle GPU time or queueing, affecting average utilization and effective cost.
• Cold Starts: The cost of loading a model into memory for the first request, which must be amortized across subsequent tokens.
• Networking & Orchestration: Overhead from load balancers, API gateways, and container orchestration (e.g., Kubernetes) adds to the total cost of service.

The choice of base model is the primary determinant of cost per token. Small Language Models (SLMs) like Llama 3.1 8B or Phi-3-mini offer dramatically lower inference costs than larger counterparts with minimal performance loss for many enterprise tasks. For specialized domains, consider Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA to adapt a smaller, general model instead of deploying a massive, underutilized one. Architectures with optimized attention mechanisms, such as Grouped-Query Attention (GQA) or Sliding Window Attention, reduce memory bandwidth pressure, directly lowering compute cost.

Compression reduces the computational footprint of a model, directly lowering the cost per token. Key techniques include:

• Model Quantization: Reducing numerical precision from FP32 to INT8 or FP8 cuts memory use and accelerates computation on supported hardware. GPTQ and AWQ are advanced algorithms for high-accuracy 4-bit quantization.
• Model Pruning: Removing redundant neurons or weights shrinks the model size, decreasing the FLOPs required per token.
• Knowledge Distillation: Training a compact student model to mimic a larger teacher model achieves comparable quality with far fewer parameters. These techniques are often combined for maximum effect.

Serving infrastructure efficiency dictates how many tokens you generate per dollar of compute. Implement:

• Continuous Batching (In-flight Batching): Dynamically batches incoming requests of different lengths, eliminating GPU idle time and maximizing throughput.
• PagedAttention & vLLM: The vLLM serving engine uses the PagedAttention algorithm to manage the KV Cache with minimal memory waste, allowing higher concurrency on the same hardware.
• Speculative Decoding: Uses a small, fast draft model to propose token sequences, which are then verified in parallel by the main model, reducing latency and effective compute time per output token.

Operational policies prevent resource waste and smooth demand. Key strategies are:

• Autoscaling: Dynamically scales the serving cluster up or down based on request queue depth, preventing costly over-provisioning during low-traffic periods.
• Rate Limiting & Load Shedding: Protects the system from traffic spikes. Load shedding rejects low-priority requests under extreme load to maintain Service Level Agreements (SLAs) for critical users.
• Request Optimization: Encourage users to employ system prompts and few-shot examples to reduce output token length and retries. Implement caching for frequent, deterministic queries.

Financial operations practices bring accountability to cloud spend. Essential activities include:

• Cloud Cost Allocation: Use resource tagging to attribute GPU instance costs to specific projects, teams, or API keys. This creates visibility into which workloads drive the highest inference cost.
• Instance Right-Sizing: Continuously analyze metrics like Tokens Per Second (TPS) and GPU utilization to select the most cost-effective instance type (e.g., comparing A100, H100, L4).
• Budget Alerts & Forecasting: Set alerts based on cost-per-token trends and use historical token consumption to forecast future spend, enabling proactive budget management.

You cannot optimize what you do not measure. Establish granular monitoring:

• Track cost per token segmented by model, endpoint, and user. Correlate it with quality metrics (e.g., evaluation scores) to avoid optimizing cost at the expense of value.
• Monitor Tail Latency (P95/P99) alongside throughput. A strategy that lowers cost but increases latency variance may degrade user experience.
• Analyze the ratio of input to output tokens, as some providers charge for both. Optimizing prompts to be concise can have a direct cost impact.

Inference cost is the total financial expenditure of running a trained model to make predictions. While cost per token is a key component, inference cost is a broader metric that aggregates all operational expenses:

• Compute resources (GPU/CPU time)
• Memory (RAM and VRAM for KV Cache)
• Networking and data transfer
• Serving infrastructure overhead It is typically measured in cost per request or cost per token and is the primary target for optimization efforts by CTOs and FinOps teams.

Tokens Per Second (TPS) is the primary throughput metric for LLM serving, measuring the number of output tokens a system can generate per second. It has a direct, inverse relationship with cost per token:

• Higher TPS means more tokens are produced for the same fixed compute cost, lowering the effective cost per token.
• Optimization techniques like dynamic batching, quantization, and efficient attention (e.g., PagedAttention) aim to maximize TPS. For high-volume applications, engineering decisions are often a trade-off between maximizing TPS and managing other costs like tail latency.

Model quantization is a compression technique that reduces the numerical precision of a model's weights and activations (e.g., from 32-bit to 8-bit integers). This is a primary method for reducing cost per token by decreasing the computational and memory footprint:

• INT8 Quantization: Reduces memory usage by ~75% vs. FP32, enabling faster inference on hardware with optimized integer units.
• GPTQ & AWQ: Advanced algorithms for 4-bit weight quantization, enabling large models to run on consumer GPUs.
• Impact: Quantization directly lowers the compute cost per token, though it may require calibration or retraining to maintain accuracy.

Dynamic batching and continuous batching (in-flight batching) are serving optimizations that maximize GPU utilization, thereby reducing the effective cost per token.

• Dynamic Batching: Groups multiple requests into a single batch, adjusting size based on load.
• Continuous Batching: A more advanced form where new requests are added to a running batch as previous sequences finish, eliminating GPU idle time. These techniques amortize the fixed cost of loading the model across more tokens, significantly improving Tokens Per Second (TPS) and lowering unit economics.

The KV (Key-Value) Cache is a memory optimization for transformer inference that stores computed attention keys and values for previous tokens. PagedAttention (used by vLLM) is an algorithm that manages this cache in non-contiguous blocks.

• Efficiency: Proper KV cache management prevents redundant computation, speeding up token generation.
• Memory Fragmentation: Inefficient caching wastes GPU VRAM, limiting batch size and concurrency, which increases cost per token.
• vLLM's Impact: By reducing memory waste via PagedAttention, vLLM allows for larger batch sizes and higher throughput, directly driving down cost per token.

FinOps is the operational practice of managing cloud financial accountability. For LLMs, it provides the framework to analyze and optimize cost per token at an organizational level.

• Cloud Cost Allocation: The process of attributing GPU instance costs, networking fees, and other expenses to specific teams or projects using resource tags.
• Instance Right-Sizing: Analyzing workload patterns to select the most cost-effective cloud instance type, avoiding over-provisioning.
• Goal: These practices move cost per token from a technical metric to a business KPI, enabling data-driven decisions on model selection, optimization, and scaling.