Federated Batch Normalization

Standard Batch Normalization (BN) computes running mean and variance statistics on local mini-batches. In a federated setting, sharing these raw statistics with the server leaks information about the local data distribution. An adversary can infer sensitive attributes about a client's dataset—such as patient demographics or disease prevalence—from the batch statistics alone. Federated BN adaptations must therefore decouple normalization from raw data exposure.

A foundational adaptation where each client maintains its own private BN statistics and affine parameters (scale and shift). These parameters are never shared with the central server. Only the convolutional or linear layer weights are aggregated globally. This approach naturally handles feature distribution skew because each client's normalization layer adapts to its local data characteristics without forcing a global consensus on feature scaling.

An alternative strategy where clients share only their aggregated running means and variances with the server, which computes a global moving average. During inference, the global model uses these synchronized statistics. This method is effective when clients have similar feature distributions but risks degrading performance under severe feature distribution skew, as a single global normalization may not fit all local domains.

Advanced techniques that replace standard BN with alternatives less sensitive to domain shift:

• Group Normalization (GN): Normalizes over channel groups independently of batch size, making it robust to heterogeneous client data.
• Layer Normalization (LN): Normalizes across all features for each sample, completely removing dependency on batch statistics.
• Instance Normalization (IN): Normalizes each sample-channel pair, useful for style transfer and medical imaging harmonization. These methods eliminate the need to share any batch statistics, inherently preserving privacy.

A specialized adaptation for cross-scanner medical image analysis. Different hospitals use MRI or CT scanners from different manufacturers (Siemens, GE, Philips), introducing systematic intensity variations. Federated Instance Normalization applies per-sample normalization locally, removing scanner-specific biases before feature extraction. The shared model learns features invariant to acquisition parameters, enabling robust segmentation across institutions without sharing raw scans or scanner metadata.

A hybrid approach where the global model learns a mixture of normalization experts. Each client is assigned a weighted combination of multiple BN modules based on its data distribution. The server aggregates the weights of these modules, allowing the model to interpolate between normalization strategies. This is particularly effective in clustered federated learning scenarios where distinct groups of hospitals share internal statistical similarities but differ significantly from other groups.

The fundamental challenge that federated batch normalization addresses. In healthcare federated learning, statistical heterogeneity manifests as differences in patient demographics, disease prevalence, and imaging protocols across hospitals.

• Feature distribution skew: Different MRI scanner vendors produce varying pixel intensity distributions
• Label distribution skew: A cardiac hospital sees different condition ratios than a general clinic
• Concept drift: Clinical definitions and treatment protocols evolve differently across institutions

Federated batch normalization must handle these distributional shifts without leaking private information through batch statistics.

A specific type of non-IID data where the marginal distribution of input features differs across clients. In medical imaging, this is pervasive due to scanner-induced batch effects.

• Siemens scanners produce systematically different intensity profiles than GE scanners
• Different hospital protocols for contrast agent timing create distributional shifts
• Patient positioning variations introduce covariate shifts in feature space

Standard batch normalization that tracks local running means and variances will encode these site-specific artifacts, degrading global model performance and potentially leaking institutional information.

Techniques that explicitly minimize the distance between feature distributions of different clients, often used alongside or as alternatives to federated batch normalization.

• Maximum Mean Discrepancy (MMD) loss penalizes distributional divergence in feature space
• CORAL loss aligns second-order statistics (covariance matrices) across clients
• Adversarial domain alignment uses gradient reversal layers to learn client-invariant representations

These methods complement batch normalization by operating at the representation level rather than the activation statistics level, providing a multi-scale approach to handling non-IID data.

The goal of training a single global model that performs accurately on entirely unseen client sites. Federated batch normalization is a critical enabler of this capability.

• Domain-invariant normalization prevents the model from relying on site-specific activation statistics
• Test-time adaptation allows batch norm statistics to be recomputed on new client data without retraining
• Leave-one-domain-out evaluation validates that the model generalizes to hospitals not in the training federation

Successful federated domain generalization means a model trained across five hospitals can deploy immediately at a sixth without performance degradation.

A privacy-preserving alternative to parameter averaging that can circumvent batch normalization synchronization challenges entirely. Clients share soft label predictions on a public dataset rather than model parameters.

• Eliminates the need to synchronize batch norm statistics across clients
• Supports heterogeneous model architectures where batch norm configurations differ
• Reduces communication costs compared to transmitting full model weights

When batch normalization privacy concerns are paramount, knowledge distillation provides a complementary paradigm that naturally handles non-IID data through ensemble-style knowledge transfer.

A communication-efficient method where clients share compact class-representative vectors (prototypes) instead of full model updates. This approach naturally handles label distribution skew without requiring synchronized batch normalization.

• Prototypes are inherently normalized representations that abstract away batch-specific statistics
• Class imbalance across clients is handled by prototype weighting rather than activation normalization
• Privacy is enhanced because raw activations and batch statistics never leave the client

Prototype learning offers an architectural alternative where the normalization burden shifts from batch statistics to the distance metric used for prototype comparison.