Federated Continual Learning

The central challenge in continual learning where a model's performance on previously learned tasks degrades as it adapts to new data distributions. In federated continual learning, this is addressed through elastic weight consolidation (EWC), which identifies and protects parameters critical to prior tasks by applying a quadratic penalty on changes to those weights. Synaptic intelligence offers an alternative by tracking each parameter's contribution to past loss reductions. Memory replay techniques store a small, privacy-compliant buffer of representative samples from earlier tasks, interleaving them with new data during local training to maintain decision boundaries without requiring access to historical patient scans.

Unlike standard federated learning where all clients participate simultaneously, federated continual learning supports asynchronous client arrival—new hospitals join the network over time with their own data distributions and label ontologies. The global model must expand its capabilities without retraining from scratch. Key mechanisms include:

• Progressive neural networks that allocate new capacity for each incoming institution
• Dynamic architecture expansion where lateral connections freeze previous task weights while new modules learn novel features
• Knowledge distillation from a frozen copy of the prior global model to preserve diagnostic accuracy on legacy tasks while the active model adapts to new data streams

Federated continual learning distinguishes between two critical scenarios:

Task-Incremental Learning (Task-IL): Each new institution or data stream introduces a distinct diagnostic task with explicit task identifiers. The model knows which task it is performing at inference time, simplifying the problem to learning task-specific output heads while sharing a common feature extractor.

Domain-Incremental Learning (Domain-IL): New institutions contribute data for the same diagnostic task but with different scanner vendors, protocols, or patient demographics. The model must generalize across shifting input distributions without task labels, requiring domain-invariant feature learning and test-time adaptation strategies.

Rehearsal-based approaches store exemplars from previous tasks to interleave with new data during local training. In the federated context, this raises privacy concerns. Solutions include:

• Differentially private generative replay: Training a local generative model with DP-SGD to synthesize representative samples from prior distributions without storing real patient data
• Federated coreset selection: Collaboratively identifying a minimal set of highly informative samples that capture the feature space of previous tasks, stored with formal privacy accounting
• Feature replay: Storing only the latent representations and their labels rather than raw images, reducing both memory footprint and re-identification risk while preserving decision boundary information

Forgetting manifests differently at the global and local levels in federated continual learning:

Global Forgetting: The aggregated model loses performance on tasks from institutions that joined early in the sequence, particularly if those clients are no longer participating in training rounds. Mitigated by server-side rehearsal buffers and federated knowledge retention constraints.

Local Forgetting: Individual client models drift away from the global consensus as they adapt to their own sequential data streams. Proximal regularization terms in the local objective function penalize deviation from the most recent global model, balancing personalization with retention of collective knowledge.

Federated continual learning systems must detect when the underlying data distribution has shifted sufficiently to warrant model adaptation. Techniques include:

• Distributional drift monitoring using statistical tests on model confidence scores and prediction entropy across rounds
• Federated embedding analysis where clients share anonymized feature statistics to detect covariate shift without exposing raw data
• Performance-based triggers that initiate local adaptation when a client's validation metrics degrade below a threshold, indicating that the current global model no longer fits the local data distribution
• Gradual drift handling through exponential moving averages of model parameters rather than abrupt replacement

The primary failure mode that FCL is designed to solve. When a neural network is sequentially trained on new data distributions (e.g., a new hospital's scanner), it exhibits a drastic performance drop on previously learned tasks.

• Stability-Plasticity Dilemma: The core trade-off between retaining old knowledge and acquiring new information.
• Weight Overwriting: New gradient updates overwrite parameters critical for prior tasks.
• Mitigation in FCL: Requires local replay buffers or regularization terms that do not violate data locality.

A technique to consolidate global knowledge without transferring raw model weights. Instead of averaging parameters, clients share logits (soft labels) on a public, unlabeled dataset.

• Proxy Dataset: A non-sensitive, publicly available medical imaging dataset used for knowledge transfer.
• Teacher-Student Dynamic: The global model acts as a teacher, and local models distill its behavior.
• Privacy Advantage: Avoids sharing model weights, reducing the attack surface for model inversion.

Strategies to approximate experience replay within the strict confines of federated privacy. Since raw patient scans cannot be stored indefinitely, generative models are used.

• Generative Replay: A local GAN or VAE learns to synthesize past data distributions for interleaved training.
• Feature Replay: Storing latent feature statistics rather than raw pixels to comply with anonymization standards.
• Constraint: The generative model must itself be trained in a privacy-preserving manner to avoid memorization.

A foundational regularization strategy adapted for FCL. EWC identifies synaptic importance by calculating the Fisher Information Matrix on previous tasks.

• Quadratic Penalty: Applies a penalty to changes in weights deemed critical for prior institutional data.
• Federated Fisher Overlap: In FCL, the global server can aggregate importance maps to identify universally critical parameters.
• Computational Cost: Calculating the Fisher diagonal is expensive but feasible for imaging models during local fine-tuning.

A progressive neural network approach where new sub-modules or lateral connections are added for each new data stream, freezing old parameters entirely.

• Hard Attention Masks: Learns binary masks over weights to isolate task-specific pathways.
• Scalability Concern: Parameter count grows linearly with the number of institutions or tasks.
• Federated Aggregation: Only the active, non-frozen parameters for the current global task are averaged, preserving historical sub-networks.

Modifications to standard Federated Averaging (FedAvg) to account for the temporal order of updates. It prevents a new institution with a large dataset from dominating the global model immediately.

• Staleness Weighting: Decays the contribution of updates from nodes that haven't participated in recent rounds.
• Temporal Smoothing: Applies momentum to the global model trajectory to resist sudden distributional shifts.
• Concept Drift Detection: Server-side monitoring to distinguish between legitimate new pathology and noisy local updates.