Federated Model Merging

Unlike standard federated averaging, model merging operates entirely in the weight space after local training is complete. Institutions fine-tune a shared foundation model on their private clinical data and share only the final delta weights or full model checkpoints. The central server then mathematically interpolates these weights using techniques like spherical linear interpolation (SLERP) or task arithmetic. This completely eliminates the need to transmit gradients or align data distributions during the aggregation phase, providing a strict privacy guarantee by decoupling the act of sharing from the act of training.

The success of weight merging relies on the empirical phenomenon of linear mode connectivity, which posits that independently fine-tuned models often reside in the same low-loss basin. Key merging strategies include:

• Spherical Linear Interpolation (SLERP): Blends two models by traversing the geodesic on the hypersphere of weights, preserving magnitude and angular velocity.
• Task Arithmetic: Adds the task-specific 'delta' vectors (fine-tuned weights minus pre-trained weights) to the base model, often scaled by a coefficient.
• TIES-Merging: Resolves interference between delta vectors by trimming low-magnitude changes, resolving sign conflicts, and merging only the aligned components.

A primary technical challenge is parameter interference, where the weight updates from different institutions conflict, causing the merged model to perform worse than the individual local models. Advanced merging algorithms actively resolve these conflicts:

• Sign Consensus: Algorithms like TIES-Merging discard weight changes where institutions disagree on the direction (sign) of the update.
• Magnitude Pruning: Low-magnitude weight deltas, which often represent noise rather than generalizable knowledge, are dropped before merging.
• Drop-and-Rescale: Redundant parameter changes are dropped, and the remaining values are rescaled to compensate, preserving the overall signal strength from the collective fine-tuning process.

Model merging drastically reduces the infrastructure burden compared to iterative federated training. The process requires only a single round of communication after local training is fully complete. There is no need for the complex server orchestration, synchronous gradient aggregation, or straggler mitigation required by standard Federated Averaging. This makes it highly suitable for cross-silo healthcare networks with limited bandwidth or strict firewall policies. Institutions can train at their own pace, and the global model is generated instantly upon receipt of the final checkpoints, enabling a 'train-then-merge' workflow.

A practical application of model merging is the creation of 'Model Soups' . Instead of selecting a single best-performing checkpoint, a server averages the weights of multiple fine-tuned models from different hyperparameter configurations or data subsets. This simple uniform averaging has been shown to improve out-of-distribution generalization and robustness without any additional training. In a federated context, this translates to averaging the final weights from diverse institutional fine-tuning runs, resulting in a global model that is often more resilient to distribution shift than any single institutional model.

While both techniques create a global model, their mechanisms differ fundamentally:

• Federated Averaging (FedAvg): Operates during training. It aggregates gradients or model updates iteratively over multiple communication rounds. Requires synchronized training and exposes intermediate model states.
• Federated Model Merging: Operates post-training. It interpolates final model weights in a single step. Training is fully asynchronous and decoupled from the aggregation server. This distinction makes merging a powerful tool for combining models that were never intended to be part of a synchronous federated learning run, such as models fine-tuned on different clinical tasks or modalities.

The core mathematical operation in model merging. Linear Interpolation (LERP) computes a weighted average of corresponding parameters from two models. Spherical Linear Interpolation (SLERP) blends weights along a hypersphere, preserving the model's weight norm and often resulting in more stable, higher-quality merges. In a federated context, these operations are performed centrally on aggregated model updates, never on raw data.

A methodology where a model's fine-tuned knowledge is isolated as a task vector—the element-wise difference between the fine-tuned weights and the original pre-trained weights. These vectors can be added, subtracted, or negated to compose model capabilities. Federated task vector merging allows institutions to contribute specific clinical competencies (e.g., radiology report generation) that are arithmetically combined into a single multi-task model.

While both combine model updates, they operate at different stages. FedAvg aggregates gradients or weight updates during training over multiple communication rounds to converge on a single global model. Model merging is a post-hoc technique that combines fully-trained, independently-fine-tuned models in a single operation without iterative communication. Merging is ideal when institutions have already specialized their models locally.

Trim, Elect Sign, and Merge. A technique designed to resolve interference between task vectors from different models. It first trims low-magnitude parameter changes to reduce noise, then elects the dominant sign for each parameter across all models to resolve directional conflicts, and finally performs a disjoint merge using only the agreed-upon parameters. This is critical in federated settings where institutions fine-tune on heterogeneous clinical data.

A pre-processing step before merging that randomly drops a high percentage (e.g., 90-99%) of delta parameters from each task vector and rescales the remaining ones to compensate. This sparsification reduces redundancy and interference between models, often leading to superior merged performance. In federated healthcare, DARE allows merging models from many institutions without catastrophic forgetting.

A simple yet effective merging approach where multiple independently fine-tuned models are averaged uniformly or with validation-set-optimized weights. Unlike complex merging algorithms, model soups require no additional training or hyperparameter tuning. In a federated context, each hospital contributes a fully-trained model, and a central server greedily soups them together, selecting combinations that maximize performance on a held-out validation set.