FedBN

The core mechanism of FedBN is that batch normalization (BN) layers are not aggregated. Each client maintains its own local running mean and variance statistics for its BN layers. During local training, a client uses its own stored statistics for normalization. This allows the model to adapt to the unique feature distribution (e.g., lighting, style, sensor calibration) of each client's data without forcing a harmful consensus on these statistics.

FedBN specifically addresses covariate shift and feature distribution shift, a common form of non-IID data in real-world federated settings. Examples include:

• Medical imaging from different hospitals with varying scanner protocols.
• Autonomous vehicles operating in distinct geographic regions with different weather and lighting.
• Mobile keyboards used by populations with different linguistic styles. By not averaging BN statistics, the model's feature normalization is tailored to each client's environment, preventing performance degradation caused by conflicting statistical signals.

The server aggregation step in FedBN is modified. Only non-BN parameters (e.g., weights from convolutional or linear layers) are averaged using standard Federated Averaging (FedAvg). The parameters of the BN layers (scale gamma and shift beta) are also typically aggregated, as they represent learned affine transformations. However, the critical statistical moments (mean, variance) remain exclusively local, decoupling the collaborative learning of features from the client-specific normalization of activations.

On tasks with significant feature shift, FedBN demonstrates:

• Faster convergence compared to vanilla FedAvg, as clients aren't fighting over divergent BN statistics.
• Higher final test accuracy on both local client data and global test sets.
• Reduced client drift, because the local optimization landscape is stabilized by client-appropriate normalization. Empirical results on benchmark datasets like DomainNet show accuracy improvements of over 10% in some domains compared to FedAvg.

FedBN assumes the neural network architecture contains batch normalization layers. Its benefits are most pronounced in architectures like CNNs and Vision Transformers where BN is standard. A key limitation is that it does not directly address label distribution skew (concept shift), which requires other personalization techniques. The method also introduces slight additional client-side memory overhead to store local BN statistics.

FedBN is a precursor to many personalized federated learning methods. Its principle—keeping certain parameters client-specific—inspires approaches like FedPer (personalized classifier heads) and FedRep (personalized representations). It establishes a hybrid model where a global feature extractor is shared, but the interface to client-specific data distributions is localized, balancing collaboration with adaptation.

The core challenge FedBN is designed to address. In federated learning, statistical heterogeneity means the data distributions across clients are not Independent and Identically Distributed (non-IID). This manifests as:

• Feature distribution shift: The same feature (e.g., pixel brightness in images) has different statistical properties (mean, variance) per client.
• Label distribution skew: The prevalence of certain classes varies drastically between clients.
• Concept shift: The relationship between features and labels differs. FedBN mitigates the effects of feature distribution shift by localizing batch normalization statistics.

A direct consequence of statistical heterogeneity where local client models diverge from the global objective during local training. This occurs because each client's SGD updates point toward the optimum of its local data distribution, not the global one. Effects include:

• Slowed or unstable convergence of the global model.
• Final model performance degradation.
• FedBN reduces client drift caused by feature distribution shifts by preventing the aggregation of local normalization parameters, which would otherwise pull the global model in conflicting directions.

A family of techniques aiming to produce models tailored to individual clients' data distributions, rather than a single global model. FedBN is considered a lightweight PFL method. Key approaches include:

• Personalization Layers: Methods like FedPer keep final model layers local.
• Meta-Learning: Framing FL as a meta-learning problem to learn easily adaptable models.
• Model Interpolation: Training a global model alongside local fine-tuned models. FedBN provides personalization implicitly through local batch norm stats, adapting the feature representation per client without changing the core model architecture.

A federated optimization algorithm that mitigates client drift by adding a proximal term to the local client loss function. The modified objective is: L_local(w) + (μ/2) * ||w - w_global||^2.

• Mechanism: The proximal term penalizes local updates that stray too far from the global model, acting as a regularizer.
• Contrast with FedBN: FedProx operates on the entire parameter vector via a regularization penalty. FedBN addresses a specific source of drift (feature shift) by architectural modification, leaving other parameters unconstrained. They can be complementary techniques.

Stochastic Controlled Averaging for Federated Learning uses control variates (correction terms) to correct for client drift. Each client maintains a control variate estimating the difference between its local and global gradient direction.

• Update Correction: Clients use their control variate to adjust their local gradient steps before sending updates.
• Server Correction: The server also maintains a global control variate.
• Comparison: SCAFFOLD corrects drift in the update direction. FedBN corrects drift in the feature representation space. SCAFFOLD is algorithm-centric; FedBN is architecture-centric.

The standard layer FedBN modifies. BatchNorm normalizes the activations of a layer for each mini-batch during training using the batch's mean and variance, then applies a learned affine transformation (scale γ and shift β).

• Standard Practice: In centralized training, γ and β are learned; the running mean/variance are global.
• FedBN Modification: In federated learning, FedBN keeps all BatchNorm parameters (γ, β, running mean, running variance) local and non-aggregated. Only non-BatchNorm parameters are averaged by the server. This simple change provides robustness to feature shift.