Weight Averaging

A procedure that averages weights from multiple checkpoints collected during the final epochs of training with a cyclical or constant learning rate schedule. Unlike traditional ensembling, SWA produces a single model with no inference overhead.

• Checkpoints are collected at the end of training, not throughout
• Uses a high constant or cyclical learning rate to explore the loss landscape
• The averaged solution sits in a wider basin of attraction, improving generalization
• Particularly effective when fine-tuning on small medical imaging datasets

A technique that averages the weights of multiple fine-tuned models, each trained with different hyperparameter configurations, to achieve superior accuracy without any additional inference cost. Unlike SWA, the models are trained independently rather than sequentially.

• Greedy soup: sequentially adds models to the average only if validation accuracy improves
• Uniform soup: simply averages all available fine-tuned models
• Works because fine-tuned models lie in a linearly connected region of the loss landscape
• Demonstrated to boost accuracy on medical imaging benchmarks like CheXpert and MIMIC-CXR

A running average of model weights maintained throughout training, where recent iterations receive exponentially higher weight than older ones. EMA produces a smoothed model that is more stable and often generalizes better than the final raw checkpoint.

• Maintained as a shadow copy of the model during training
• The decay rate controls how much historical weights influence the average
• Commonly used in self-supervised pre-training for medical images
• Provides a form of temporal ensembling without storing multiple checkpoints

Weight averaging exploits the geometry of neural network loss surfaces. Fine-tuned solutions often reside in the same wide, flat basin of the loss landscape, making linear interpolation between them viable.

• Flat minima generalize better than sharp minima because small weight perturbations don't degrade performance
• Averaging acts as an implicit sharpness-aware regularization
• Visualized via loss contour plots showing averaged solutions at the center of wide basins
• Explains why averaging weights from different fine-tuning runs yields robust diagnostic models

Weight averaging is straightforward to implement during fine-tuning of pre-trained models on medical imaging tasks. The technique requires minimal code changes and integrates with existing training loops.

• SWA: collect checkpoints at the end of each epoch during the final phase of fine-tuning
• EMA: update a shadow model after each optimizer step using a decay factor (e.g., 0.999)
• Model Soup: independently fine-tune with varied learning rates or augmentations, then average
• Compatible with LoRA and adapter-based fine-tuning methods
• Particularly valuable when annotated medical data is scarce, as it reduces overfitting

Weight averaging provides the generalization benefits of model ensembling without the linear increase in inference cost. This is critical for deploying diagnostic models in resource-constrained clinical environments.

• Ensembling: requires N forward passes for N models; averages predictions
• Weight averaging: single forward pass; averages parameters before inference
• Both techniques reduce variance in predictions
• Weight averaging is preferred for edge deployment of diagnostic AI where compute is limited
• Can be combined with test-time augmentation for further robustness gains

A direct application of weight averaging where multiple fine-tuned models with different hyperparameter configurations are averaged into a single model. Unlike traditional ensembles that require running multiple models at inference, a model soup collapses diversity into one set of weights, providing accuracy gains with zero additional inference cost. Popularized by the original paper demonstrating ImageNet accuracy improvements without any runtime overhead.

A weight averaging technique that maintains a decaying average of model parameters throughout training, giving more weight to recent updates. EMA models are commonly used as the final evaluation checkpoint because they exhibit smoother convergence and better generalization than the raw training trajectory. Widely adopted in self-supervised learning frameworks like BYOL and MoCo for medical image pre-training, where stable target representations are critical.

The phenomenon where a neural network abruptly overwrites previously learned knowledge when fine-tuned on a new target task. Weight averaging helps mitigate this by interpolating between the pre-trained and fine-tuned weights, preserving source domain representations. Key mitigation strategies include:

• Elastic Weight Consolidation (EWC): Penalizes changes to important parameters
• Progressive Networks: Freeze old tasks and add lateral connections
• Weight Interpolation: Average pre-trained and fine-tuned weights to balance old and new knowledge

The challenge of training a model on one or several source domains that must robustly generalize to entirely unseen target domains without any adaptation. Weight averaging contributes to domain generalization by finding flatter minima that are less sensitive to distribution shifts. In medical imaging, this is critical for deploying models that work across:

• Different MRI scanner manufacturers (Siemens, GE, Philips)
• Varying acquisition protocols and field strengths
• Unseen patient demographics and disease presentations

A technique that updates a model's normalization statistics or parameters during inference on target samples to combat distribution shift. Unlike weight averaging which is applied offline, test-time adaptation operates dynamically. Common approaches:

• BatchNorm recalibration: Update running statistics on target data
• Entropy minimization: Adjust weights to produce confident predictions
• TENT: Optimize channel-wise affine parameters via entropy minimization

Weight-averaged models provide a more robust initialization for test-time adaptation, reducing the risk of destabilization.