Model Soups

The simplest instantiation of a model soup involves taking the element-wise arithmetic mean of the weights of multiple fine-tuned models. Each model is typically initialized from the same pre-trained checkpoint but fine-tuned with distinct hyperparameter configurations (e.g., varying learning rates, data augmentations, or random seeds).

• Mechanism: For N models, the final weight is θ_soup = (1/N) * Σ θ_i.
• Key Insight: This works because fine-tuned models often reside in a single low-error basin in the loss landscape; averaging their weights finds a central point that is more robust than any individual solution.
• Result: The uniform soup often outperforms the best individual model on out-of-distribution test sets without any increase in inference latency.

A more sophisticated approach where models are sequentially added to the soup only if their inclusion improves validation accuracy. This prunes models that have diverged into incompatible loss basins.

• Algorithm: Sort candidate models by validation accuracy. Starting with the best model, iteratively add the next model to the running average; keep the addition only if validation performance increases.
• Benefit: Prevents negative interference from outlier models that fine-tuned into distinct, incompatible minima.
• Use Case: Essential when fine-tuning across a wide hyperparameter sweep where some runs may collapse or overfit severely.

Model soups exploit the property of linear mode connectivity—the empirical finding that minima reached by fine-tuning from a shared pre-trained initialization are often connected by a path of low loss in weight space.

• Linear Interpolation: Moving along the line segment θ_α = (1-α)θ_1 + αθ_2 between two fine-tuned weights maintains high accuracy for all α ∈ [0,1].
• Barrier-Free Basins: Unlike training from scratch, fine-tuning constrains solutions to a shared basin, making weight averaging viable.
• Implication: The soup's averaged weight is not an arbitrary midpoint but a legitimate, high-performing solution in its own right.

The primary advantage of model soups over single models is enhanced out-of-distribution (OOD) generalization. By averaging weights, the soup smooths out the sharp, brittle minima that individual models may converge to.

• Sharp vs. Flat Minima: Individual fine-tuned models can settle into sharp minima that fit the training distribution perfectly but fail catastrophically under covariate shift. Averaging steers the solution toward flatter minima.
• Empirical Gains: On benchmarks like ImageNet and its distribution-shift variants (ImageNet-V2, ImageNet-R, ImageNet-Sketch), model soups consistently improve accuracy by 0.5-1.5% absolute over the best single model.
• Medical Imaging Relevance: This property is critical for diagnostic models deployed across different hospital scanners, where domain shift is the primary failure mode.

Unlike traditional output-space ensembles that require running multiple forward passes and averaging logits, a model soup produces a single set of weights. This collapses the ensemble cost to that of a single model.

• Traditional Ensemble Cost: Running K models requires K times the FLOPs and memory.
• Soup Cost: A single forward pass with the averaged weights. The computational cost is paid entirely upfront during the averaging step.
• Deployment Advantage: This makes model soups ideal for edge deployment and real-time diagnostic systems where latency budgets are strict and batching multiple models is infeasible.

In medical imaging, where annotated datasets are small and domain shift between scanners is severe, model soups provide a practical method to stabilize fine-tuning runs.

• Hyperparameter Sensitivity: Fine-tuning on small medical datasets is notoriously brittle to learning rate and augmentation choices. A soup over a hyperparameter sweep mitigates this fragility.
• Cross-Scanner Generalization: Averaging models fine-tuned with different Hounsfield Unit normalization windows or intensity augmentations creates a model robust to scanner variability.
• Synergy with LoRA: Combining model soups with Low-Rank Adaptation allows averaging only the lightweight adapter weights, making the technique computationally trivial while preserving the base model's integrity.

The foundational mechanism behind Model Soups. Instead of selecting a single best checkpoint, weight averaging interpolates between multiple points along the fine-tuning trajectory to find a flatter loss minimum. This is distinct from traditional output-space ensembling—it produces a single model with no inference overhead. In medical imaging, averaging weights from epochs 45, 50, and 55 of a segmentation model often yields better Dice scores than any individual checkpoint.

Addresses the domain shift problem that Model Soups help mitigate. When source and target distributions differ—such as training on natural images but deploying on CT scans—domain adaptation techniques align feature spaces. Model Soups complement this by:

• Averaging models fine-tuned with different adaptation strengths
• Smoothing out adaptation artifacts from aggressive domain alignment
• Improving robustness to residual domain gaps across scanner vendors Combined with CycleGAN adaptation or domain-adversarial training, Soups produce more reliable cross-domain performance.

The upstream process that creates the initial weights later refined into Model Soups. Self-supervised pre-training learns representations from unlabeled medical images using pretext tasks:

• Contrastive learning: Pulls augmented views of the same image together
• Masked Image Modeling (MIM): Reconstructs hidden patches, popular with Vision Transformers Models pre-trained via anatomical pre-training on organ segmentation tasks provide stronger initialization for Soup construction than generic ImageNet weights, as the learned features are clinically relevant.

The failure mode that Model Soups help prevent. During fine-tuning, a network can abruptly overwrite previously learned general features, losing the representational power that made pre-training valuable. Model Soups counteract this by:

• Averaging early checkpoints (strong general features) with later ones (task-specific adaptation)
• Creating a compromise between stability and plasticity
• Reducing sensitivity to the exact epoch where forgetting begins In medical imaging, this is critical—a model must retain anatomical understanding while learning pathology-specific patterns.