GGUF Format

A GGUF file encapsulates the entire model—including the hyperparameters, tokenizer vocabulary, and quantized weight tensors—into a single, portable binary. This eliminates the dependency on scattered configuration files (like config.json or tokenizer.json), drastically simplifying version control and distribution. The format uses a key-value metadata header to store all necessary configuration, ensuring the inference engine has everything it needs to load and run the model without external references.

The header of a GGUF file is a rich dictionary of key-value pairs that define the model's architecture and provenance. This replaces the brittle, unstructured text headers of GGML. Standardized keys define critical parameters like llama.context_length, llama.embedding_length, and llama.block_count, while custom keys allow for community-driven extensions. This structured metadata enables the llama.cpp runtime to automatically configure itself for a model without hardcoded logic for each architecture variant.

GGUF is purpose-built to store quantized tensors efficiently. It supports a wide range of quantization types directly in the file format, including k-quant strategies (e.g., Q4_K_M, Q5_K_M) that intelligently mix block sizes for critical layers. The format stores the quantization parameters alongside the tensor data, allowing the inference engine to dequantize weights on-the-fly during computation. This tight coupling between storage and computation is a core reason for the format's speed advantage.

The GGUF format is designed for long-term stability through a robust versioning system. The file header specifies a format version, allowing the llama.cpp loader to support multiple GGUF versions simultaneously. This ensures that older, quantized models do not become instantly obsolete when the format is updated. The extensible metadata system also allows new features to be added without breaking existing parsers, as unknown keys are simply ignored by older runtimes.

GGUF files are structured to enable memory mapping (mmap) directly from disk. Instead of reading the entire multi-gigabyte file into RAM before inference begins, the operating system maps the file into virtual memory. The inference engine then pages in only the required weight tensors on demand. This results in near-instant model loading times and allows multiple processes to share the same physical memory pages for the model weights, dramatically reducing overall system RAM consumption.

Internally, GGUF aligns tensor data blocks to specific byte boundaries. This alignment is a critical optimization for modern CPUs, as it allows the inference engine to perform SIMD (Single Instruction, Multiple Data) operations without performance penalties caused by unaligned memory access. By ensuring that weight matrices start at aligned addresses, the format directly contributes to the raw token generation speed that makes llama.cpp competitive on consumer hardware without dedicated accelerators.

The core compression technique that makes GGUF files practical. Quantization reduces the numerical precision of weights from 16-bit floating point to lower bit widths like 4-bit or 5-bit integers. This dramatically shrinks model size and memory usage while preserving output quality. GGUF files embed quantization parameters directly in their metadata, enabling the llama.cpp runtime to dequantize weights on-the-fly during inference without separate calibration files.

The memory buffer that stores computed Key and Value tensors from previous tokens during autoregressive generation. Without caching, each new token would require recomputing attention over the entire sequence. GGUF-compatible runtimes manage the KV cache in quantized formats to reduce memory pressure. Techniques like PagedAttention partition the cache into non-contiguous blocks, eliminating fragmentation and enabling high-throughput serving of long-context models.

The maximum number of tokens a model can process in a single forward pass, defined in the GGUF metadata header. This parameter determines the upper limit of the model's immediate working memory. GGUF files specify the native context length from training, but runtimes can apply RoPE scaling techniques to extend this without retraining. Practical considerations:

• Larger windows consume proportionally more KV cache memory
• Quantized KV caches trade precision for extended context
• Context shifting in llama.cpp enables efficient sliding-window attention

The process of segmenting raw text into discrete tokens from a predefined vocabulary, stored within the GGUF file's metadata. GGUF embeds the tokenizer model directly, ensuring the inference runtime uses the exact same tokenization as training. Most GGUF models use Byte-Pair Encoding (BPE) or SentencePiece subword tokenizers. The tokenizer configuration includes:

• Vocabulary mappings
• Special token IDs (BOS, EOS, PAD)
• Merge rules for BPE
• Pre-tokenization regex patterns