Llama.cpp

The native binary file format for llama.cpp, designed for single-file deployment of quantized models. GGUF stores the model's hyperparameters, tokenizer vocabulary, and quantized weight tensors in a self-contained structure. Key advantages include:

• Backward compatibility through a key-value metadata system
• Support for dozens of quantization types (Q4_0, Q5_K_M, IQ3_XXS, etc.)
• Memory-mapped loading for near-instant model initialization
• Native support for split models across multiple files for easier distribution

The compression technique that makes llama.cpp viable on consumer hardware. Quantization reduces weight precision from 16-bit floating point to as low as 2 or 3 bits per parameter while preserving output quality. llama.cpp supports:

• K-quant strategies (Q4_K_M, Q5_K_M) that apply mixed precision to different layers
• I-quant types (IQ2_XXS, IQ3_XXS) using importance-based quantization for extreme compression
• Perplexity benchmarks showing Q4_K_M achieves near-lossless quality at 4.5 bits per weight A 70B parameter model quantized to 4 bits fits in ~40GB RAM instead of 140GB.

The key-value cache stores attention tensors from previously processed tokens to avoid recomputation during autoregressive generation. llama.cpp implements:

• FP16 and quantized cache options (Q8_0, Q4_0) to reduce memory pressure
• Cache shifting for context extension beyond the original training length
• Prompt caching that reuses KV states for identical system prompts across requests
• Flash Attention integration for memory-efficient exact attention computation Efficient KV cache management is critical for long-context inference on memory-constrained devices.

A user-friendly wrapper built on top of llama.cpp that simplifies local LLM deployment. Ollama provides:

• A Modelfile system for reproducible model configuration and prompt templates
• REST API for programmatic access with OpenAI-compatible endpoints
• Automatic model pulling from a curated library of pre-quantized GGUF models
• GPU acceleration detection and configuration without manual setup
• Multi-model management with versioning and concurrent serving Ollama abstracts away the complexity of manual quantization selection and CLI flags, making llama.cpp accessible to developers who need a quick local LLM.

A serving optimization that llama.cpp's server mode implements to maximize throughput. Instead of waiting for an entire batch of sequences to complete generation before accepting new requests, continuous batching:

• Dynamically evicts completed sequences from the batch
• Inserts new requests into the running batch immediately
• Reduces idle GPU time by keeping the batch full at all times
• Works in conjunction with parallel decoding across multiple sequences This technique is essential for production deployments handling concurrent users with varying prompt lengths and generation requirements.

An inference acceleration technique supported by llama.cpp that uses a draft model to propose multiple future tokens quickly, which are then verified in parallel by the target model. Benefits include:

• 2-3x speedup in tokens-per-second on single-user workloads
• No loss in output quality — verification guarantees identical distribution
• Works with any draft-target model pair sharing the same tokenizer
• Particularly effective for code generation and structured output where token patterns are predictable llama.cpp implements speculative decoding through its --draft-model parameter, enabling faster local inference without hardware upgrades.