Federated Learning with Model Heterogeneity

The primary driver of model heterogeneity is the vast disparity in computational resources across the federated network. Clients range from powerful servers to smartphones and microcontrollers. This necessitates local models with different:

• Parameter counts (e.g., 100M vs. 1M parameters)
• Architectural complexity (e.g., ResNet-50 vs. MobileNetV2)
• Memory footprints and inference latency targets A single global architecture is infeasible, forcing a paradigm where the server must aggregate knowledge from fundamentally different model structures.

Clients often have unique local objectives despite participating in a broader collaborative task. For example, in a federated visual assistant:

• A smartphone camera app may need a model optimized for portrait mode and HDR.
• A security camera may require a model specialized for low-light object detection.
• A drone may need a model for aerial image segmentation. This functional divergence requires specialized model architectures (e.g., different convolutional heads, attention mechanisms) that cannot be served by a monolithic global model, directly driving architectural heterogeneity.

Clients frequently possess data from different sensors or sources, leading to incompatible input feature spaces. This is a strict architectural constraint. Examples include:

• Medical imaging: One hospital uses MRI scans (3D volumes), another uses X-rays (2D grayscale), and a third uses dermatology photos (2D RGB).
• IoT networks: Vibration sensors, temperature gauges, and acoustic monitors all produce temporally aligned but structurally distinct data streams. These clients must employ entirely different input layers and early-stage feature extractors, making parameter-averaging aggregation methods like FedAvg impossible and necessitating heterogeneous-model federated learning techniques.

Strict data sovereignty regulations (e.g., GDPR, HIPAA) and corporate policies can mandate that certain layers of a model must remain exclusively on-premises or on-device. This creates a split-model paradigm. A common pattern is:

• Private local layers: The initial layers that process raw, sensitive data (e.g., medical records, financial transactions) remain on the client and are never shared.
• Shareable abstract layers: Only the outputs from higher, more abstract layers (e.g., embeddings, logits) are sent to the server for aggregation. This architectural partitioning is a deliberate design choice that institutionalizes model heterogeneity across the federation.

Transmitting full model updates from large architectures is prohibitive for clients with metered or low-bandwidth connections (e.g., satellites, rural IoT). This drives the adoption of:

• Heterogeneous compression: Clients with poor connectivity may use extreme pruning or quantization, producing a structurally different, sparse model compared to a full-precision model from a well-connected client.
• Knowledge distillation targets: Instead of sharing parameters, clients may share soft labels or embeddings, which are architecture-agnostic. The server then distills this knowledge into a variety of client-side model sizes, perpetuating heterogeneity. The need for communication efficiency thus actively encourages, rather than merely tolerating, model diversity.

In enterprise and industrial settings, federated learning must integrate with existing, deployed inference models. Retraining or replacing these legacy models is often cost-prohibitive or operationally risky. Drivers include:

• Vendor lock-in: Proprietary models from different equipment manufacturers.
• Certification overhead: Models in regulated industries (aviation, healthcare) require lengthy re-certification for any architectural change.
• Technical debt: Models embedded in firmware or specialized hardware (ASICs, FPGAs). The federation must therefore develop aggregation methods (e.g., federated distillation, logit averaging) that can improve a collection of fixed, heterogeneous architectures without altering their core structure.

A family of techniques designed to produce models tailored to individual clients' local data distributions, rather than a single global model. This directly addresses performance disparities caused by Non-IID data.

• Core Methods: Include training local models with regularization towards a global model (e.g., Ditto), meta-learning approaches, and multi-task learning frameworks.
• Contrast with Model Heterogeneity: While PFL often assumes a shared architecture, model heterogeneity allows architectures themselves to differ, adding another layer of complexity to personalization.

A federated optimization algorithm that modifies the local client objective by adding a proximal term. This term penalizes updates that stray too far from the current global model, thereby mitigating client drift caused by statistical and system heterogeneity.

• Mechanism: The local loss function is L(w) + (μ/2) * ||w - w^t||², where w^t is the global model and μ is a hyperparameter.
• Relevance to Model Heterogeneity: Provides a stabilizing force during local training, which is especially critical when clients have different model architectures and are thus more prone to divergent optimization paths.

Stochastic Controlled Averaging for Federated Learning. This algorithm uses control variates—client-specific and global correction terms—to correct for client drift, reducing the variance between local updates.

• How it Works: Each client maintains a state variable (control variate) that estimates the drift between its local updates and the global direction. This state is used to correct subsequent updates.
• Connection to Heterogeneity: SCAFFOLD's variance reduction is crucial when clients have heterogeneous data and models, as the inherent update directions will be more diverse, complicating aggregation.

An approach where clients are grouped into clusters based on data or model similarity, and a separate global model is trained for each cluster. This handles Non-IID data by creating multiple, more homogeneous model groups.

• Implementation: Clustering can be based on model update similarities, loss landscapes, or data distributions.
• Synergy with Model Heterogeneity: This paradigm naturally accommodates model heterogeneity by allowing different clusters to potentially use different optimal architectures suited to their members' data and hardware.

A decentralized training paradigm where clients train local models and share knowledge (e.g., logits, soft labels, or embeddings) instead of model parameters. This can be more communication-efficient and robust to Non-IID data.

• Process: A server aggregates knowledge (like average logits for a set of data samples) and distills it back to clients to refine their local models.
• Advantage for Heterogeneous Models: Since clients only share outputs, not parameters, they can use completely different neural network architectures, making FD a foundational technique for heterogeneous federated learning.

A technique where clients keep their local batch normalization (BN) layers private and do not share their statistics (mean and variance) during aggregation. This helps mitigate feature shift caused by Non-IID data.

• Rationale: BN statistics are highly sensitive to local data distribution. Aggregating them can harm performance. FedBN treats BN layers as local parameters.
• Relation to Model Heterogeneity: This is a specific, critical instance of partial model sharing, a common strategy in heterogeneous FL where only a subset of layers (excluding BN) are aggregated.