Model Checkpointing

The learned parameters of the neural network, typically stored as multi-dimensional tensors. These are the numerical values that define the model's function.

• Format: Often saved in .pt, .pth (PyTorch), .h5, .pb (TensorFlow), or Safetensors format
• Size: Can range from megabytes for small models to hundreds of gigabytes for large language models
• Precision: May be stored in FP32, FP16, or BF16 depending on training configuration
• Sharding: Large models split weights across multiple files for distributed storage and loading

The internal variables maintained by the optimization algorithm, such as momentum buffers and adaptive learning rate accumulators. Without this, training cannot resume seamlessly.

• Adam/AdamW: Stores first-moment (m) and second-moment (v) estimates for each parameter
• SGD with Momentum: Preserves velocity vectors that smooth gradient updates
• Memory Cost: Optimizer state often consumes 2-3x more memory than the model weights themselves
• Resumption: Omitting optimizer state forces a cold restart of optimization dynamics, potentially disrupting convergence

Non-tensor information that tracks the progress and configuration of the training run, essential for reproducibility and debugging.

• Global Step / Epoch: The exact iteration count at which the checkpoint was saved
• Learning Rate Schedule: Current learning rate and scheduler state (e.g., cosine annealing phase)
• Random Seed State: The state of all random number generators (Python, NumPy, PyTorch) for exact reproducibility
• Loss and Metrics: Accumulated training and validation metrics up to the save point
• Data Loader State: The position in the dataset iterator for resuming data feeding without skipping or repeating samples

The structural blueprint of the neural network, including layer types, connectivity patterns, and hyperparameters. Weights are meaningless without this schema.

• Serialized Graph: A representation of the computational graph (e.g., model.state_dict() in PyTorch paired with the model class definition)
• Configuration File: A .yaml or .json file specifying layer dimensions, activation functions, dropout rates, and other architectural choices
• Versioning: Architecture changes between checkpoints must be tracked; loading weights into an incompatible architecture causes shape mismatch errors
• Best Practice: Store the full model definition code or a serialized version (TorchScript, ONNX) alongside weights to avoid dependency on external codebases

For NLP and multimodal models, the tokenizer vocabulary and preprocessing parameters are a critical part of the checkpoint. A mismatch between training and inference tokenization silently corrupts inputs.

• Vocabulary Files: Byte-pair encoding (BPE) merges, WordPiece vocabularies, or SentencePiece models
• Special Token IDs: Mappings for [CLS], [SEP], [PAD], [UNK], and custom tokens
• Normalization Rules: Lowercasing, Unicode normalization forms (NFC, NFD), and stripping rules
• Image Preprocessing: For vision models, mean/std normalization values, resize dimensions, and augmentation pipelines must be preserved

The policy governing when and how checkpoints are saved, balancing storage costs against recovery granularity.

• Periodic Checkpointing: Save every N steps or every T minutes, providing regular recovery points
• Top-K Checkpointing: Retain only the K checkpoints with the best validation metric, discarding inferior ones
• Fault-Tolerant Checkpointing: Asynchronous saves that do not block training, using techniques like torch.save with non-blocking I/O
• Distributed Checkpointing: In multi-GPU training, coordinating saves across ranks to avoid corruption; frameworks like PyTorch Distributed Checkpoint (DCP) handle this atomically
• Incremental Checkpointing: Only save the delta from the previous checkpoint to reduce I/O overhead in large-scale training