Stochastic Weight Averaging (SWA)

SWA guides the optimizer toward flat minima rather than sharp, narrow basins. Flat minima are regions where small perturbations to the weights cause minimal change in loss, leading to superior generalization on unseen data.

• Mechanism: Averages weights from multiple points along the SGD trajectory using a cyclical or constant learning rate
• Result: The averaged solution sits in the center of a wide basin, making it robust to the shift between training and test loss landscapes
• Key insight: Sharp minima often memorize noise; flat minima capture invariant patterns

SWA approximates the behavior of a deep ensemble using a single set of weights, achieving ensemble-like calibration without the computational burden of multiple forward passes.

• How it works: Each weight checkpoint along the trajectory represents a distinct functional mode; averaging them fuses diverse predictions into one model
• Calibration benefit: The weight averaging smooths out overconfident predictions, naturally reducing Expected Calibration Error (ECE)
• Inference cost: Identical to a standard model — no additional latency or memory overhead

SWA employs a distinct two-phase training protocol that separates standard convergence from the averaging collection phase.

• Phase 1 — Standard Training: Train the model conventionally with a decaying learning rate until convergence, typically for 75% of the total budget
• Phase 2 — SWA Collection: Continue training with a constant or cyclical high learning rate, capturing model weights at each epoch or every k iterations
• Final model: Computed as the element-wise mean of all collected weights from Phase 2
• Batch normalization: Requires a forward pass over the training data to recompute running statistics after averaging

SWA inherently produces well-calibrated probability estimates, often matching or exceeding the performance of dedicated calibration techniques like temperature scaling.

• Why it works: The weight averaging process smooths the decision boundary, preventing the extreme logit magnitudes that cause overconfidence
• Empirical results: On CIFAR-100 with Wide ResNet-28-10, SWA reduces ECE from 8.6% to 3.1% without any post-hoc adjustment
• Complementary: Can be combined with label smoothing or temperature scaling for further calibration gains

SWA exploits the geometry of the loss landscape, specifically the tendency of SGD with constant learning rates to orbit wide basins rather than descending into sharp crevices.

• Cyclical learning rates: Cause the optimizer to bounce around the rim of flat minima, sampling diverse high-performing weight configurations
• Averaging effect: The mean of these samples lies deeper in the basin than any individual point, often achieving lower test loss
• Connection to FGE: Builds on Fast Geometric Ensembling, which showed that high-learning-rate SGD trajectories connect modes via curves of low loss

Implementing SWA requires attention to batch normalization statistics and the choice of learning rate schedule during the collection phase.

• BN update: After weight averaging, run a single forward pass over a subset of training data to recompute running mean and variance for all BatchNorm layers
• Learning rate choice: A constant learning rate equal to the final value from Phase 1, or a cyclical cosine schedule, both work effectively
• Checkpoint frequency: Collecting weights once per epoch is standard; higher frequencies provide marginal gains at increased storage cost
• Framework support: Native implementations available in PyTorch (torch.optim.swa_utils) and TensorFlow Addons

The primary metric for measuring how well a model's confidence scores align with its actual accuracy. ECE partitions predictions into M equally-spaced bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin. A perfectly calibrated model achieves an ECE of zero. SWA's flatter minima often yield lower ECE scores because the averaged weights produce less overconfident predictions on out-of-distribution samples.

A post-hoc calibration method that divides all logits by a single scalar parameter T optimized on a held-out validation set. When T > 1, the softmax distribution softens, reducing overconfidence. Temperature scaling preserves accuracy while dramatically improving ECE. It pairs naturally with SWA: the averaged weights provide a better starting point for temperature optimization, as the logit magnitudes are already less extreme than those from sharp minima.

A method that trains multiple independent models with different random initializations and aggregates their predictive distributions. While SWA approximates an ensemble by averaging weights along a single trajectory, deep ensembles explicitly train separate models. Both techniques explore distinct modes of the loss landscape. SWA is computationally cheaper at inference time since it produces a single model, whereas ensembles require multiple forward passes but often achieve superior calibration and uncertainty estimates.

A regularization technique that replaces hard one-hot targets with soft targets by assigning a small probability mass ε to incorrect classes. This prevents the model from being forced to predict 100% confidence on training examples, implicitly encouraging flatter minima. When combined with SWA, label smoothing provides a complementary regularization effect: smoothing acts during training while SWA acts during the weight averaging phase, together producing models that are both accurate and well-calibrated.

A data augmentation strategy that trains on convex combinations of input pairs and their labels: x̃ = λxᵢ + (1-λ)xⱼ. This forces the model to behave linearly between training examples, reducing overconfident predictions near decision boundaries. Mixup and SWA target the same goal from different angles: mixup shapes the training landscape to have wider optima, while SWA finds and settles into those flat regions during the final epochs of training.

An inference paradigm where the model abstains from predicting when its confidence falls below a calibrated threshold. SWA improves selective classification by producing more reliable confidence estimates, making the trade-off between coverage and accuracy more favorable. The risk-coverage curve shifts downward: for any given coverage level, the model achieves lower error. This is critical for high-stakes applications where wrong predictions carry severe consequences.