Heterogeneous Federated Averaging (HeteroFA)

The core mechanism of HeteroFA is to weight each client's model update contribution based on its computational capability or the amount of work performed, rather than simply averaging all updates equally. This prevents slower, resource-constrained devices from being drowned out by more powerful clients.

• Example: A high-end server contributing 100 local training epochs might receive an aggregation weight of 1.0, while a microcontroller that managed only 5 epochs receives a weight of 0.05.
• Purpose: This ensures the global model converges in a direction informed by all device tiers, not just the fastest ones, leading to a more robust and generalizable model.

Unlike standard FedAvg which mandates a fixed number of local epochs (E) or steps, HeteroFA allows clients to perform variable amounts of local computation based on their real-time resources.

• A device with ample battery and CPU can perform many local SGD steps.
• A thermally-throttled or memory-constrained device can perform fewer steps, or even just one step, and still contribute a meaningful update.
• This elastic computation is key to accommodating dynamic resource availability on edge devices without forcing them to drop out of the training round entirely.

In a heterogeneous system, a naive aggregation that waits for all clients can be dominated by the slowest straggler devices, or conversely, can bias the model towards the fastest clients if rounds are cut short. HeteroFA explicitly addresses this.

• By weighting updates and allowing variable work, it decouples update quality from client speed.
• The server can perform aggregation once a sufficient weighted fraction of total expected work is received, rather than waiting for a fixed number of clients. This prevents the global model from being skewed by the statistical properties of whichever device tier happens to finish first.

HeteroFA is often paired with capability-aware client selection strategies. The server uses a device registry to profile clients and can strategically sample a mix of high- and low-capability devices for each round.

• Stratified sampling ensures representation across capability tiers.
• The server can predict the approximate contribution weight a selected client will provide, allowing for more accurate round planning and convergence forecasting.
• This moves federated learning from a 'participation lottery' to a resource-managed orchestration system.

HeteroFA's weighting mechanism operates on the aggregated update vector, making it orthogonal to and compatible with other critical federated learning techniques.

• Compression: Updates can be compressed (e.g., via quantization or sparsification) before being sent; the server applies the capability weight to the compressed update.
• Secure Aggregation: Cryptographic protocols that sum client updates can be modified to sum weighted updates, preserving privacy while implementing HeteroFA logic.
• Differential Privacy: Noise calibrated for privacy can be added to the weighted update, ensuring formal guarantees are maintained across heterogeneous contributions.

HeteroFA provides the mathematical basis for more advanced hierarchical aggregation schemes like tiered aggregation. In such systems:

1. Clients are grouped into tiers based on capability profiles (e.g., Tier 1: smartphones, Tier 2: IoT gateways, Tier 3: sensors).
2. Updates within a tier are averaged first (applying HeteroFA weights within the tier).
3. These tier-level models are then aggregated at the global server, potentially with tier-specific weights.

This reduces communication to the central server and allows for specialized tier-specific model adaptations, which is a step towards personalized federated learning.

A global language model is improved by learning from typing data across billions of devices. HeteroFA is critical because participants range from flagship phones to budget models.

• High-end phones perform many local epochs with large batch sizes.
• Low-end phones may perform fewer epochs or use a dynamically pruned sub-model.
• The server uses weighted averaging, scaling each device's update by its processed data volume and a capability factor, preventing slower devices from being drowned out.

Hospitals collaborate to train a medical imaging model (e.g., for detecting pathologies in X-rays) without sharing patient data. Device heterogeneity is significant.

• Research hospitals use powerful on-premise GPU clusters for local training.
• Rural clinics may use standard workstations with limited GPU memory.
• HeteroFA accommodates this via variable-length training rounds. The clinic's workstation might run 5 local epochs, while the research hospital runs 20. The server aggregates updates using tiered aggregation, first combining updates from similar capability tiers to balance influence.

A manufacturing plant trains a model to predict machine failure using sensor data from hundreds of heterogeneous devices on the factory floor.

• Newer gateways have multi-core processors and can train complex models.
• Legacy sensors have microcontrollers (TinyML devices) with kilobytes of RAM.
• HeteroFA is implemented via adaptive model partitioning. For powerful gateways, the full model is sent. For microcontrollers, only the initial feature extraction layers are deployed locally; deeper layers are computed on a nearby edge server. The server aggregates these partial updates into the global model.

An automotive company improves autonomous driving algorithms by learning from real-world data collected by cars. The fleet contains multiple model years with different onboard compute.

• Latest models have dedicated AI accelerators (NPUs).
• Older models rely on general-purpose CPUs.
• The system uses capability-based profiling and resource-aware scheduling. Training tasks are only sent to vehicles when they are parked and charging. The server implements asynchronous federated updates, accepting and aggregating gradients from fast and slow trainers as they complete, without blocking a synchronous round.

A retailer builds a recommendation model by learning from user interactions across its mobile app and in-store kiosks. The compute environment is highly diverse.

• Customer smartphones have volatile availability and varying chipset performance.
• In-store kiosks have consistent power and mid-range compute.
• HeteroFA employs stratified client sampling to ensure each training round includes a mix of device types. It also uses per-client learning rate tuning, applying a larger effective learning rate to updates from slower, less frequently available devices to increase their influence on the global model.

Researchers train acoustic models to identify species from audio recorded by a network of remote, solar-powered acoustic sensors.

• Devices are severely resource-constrained (limited CPU, battery-powered).
• Connectivity is intermittent via low-bandwidth satellite links.
• HeteroFA is adapted using connectivity-aware compression and battery-aware federated learning. Updates are heavily quantized and sparsified before transmission. The server prioritizes updates from devices with sufficient battery reserve and uses availability-aware round scheduling, aligning training with peak solar charging periods.

A participant selection method that ensures a representative mix of devices from different capability tiers (e.g., high-end phones, low-end IoT sensors) in each federated round. This mitigates selection bias where the global model could become overfit to data from only the most powerful devices.

• Purpose: To produce a model that generalizes across the entire heterogeneous device population.
• Mechanism: The server maintains a registry of client capabilities and samples participants proportionally from each predefined stratum or tier.

A communication protocol where the server aggregates client model updates as soon as they are received, without waiting for a synchronized round to finish. This is critical for systems with highly variable training times due to hardware differences.

• Contrast with SyncFL: Unlike synchronous Federated Averaging (FedAvg), it does not suffer from straggler problems.
• Aggregation: Requires techniques like weighting by staleness to handle the fact that updates are based on progressively older global model versions.

A federated protocol that allows clients to perform different numbers of local SGD steps based on their available resources, rather than enforcing a fixed epoch count. A smartphone might run 10 local epochs, while a sensor runs only 2.

• Advantage: Prevents resource-constrained devices from timing out or failing the round.
• Challenge: The server must account for this variation during aggregation, often by weighting updates by the amount of local work performed.

A technique where a random subset of neurons or entire layers is deactivated ('dropped out') for individual clients during a training round. This creates smaller, unique sub-models for each device.

• Reduces Computation: Smaller active model size lowers memory and compute load per client.
• Improves Generalization: Acts as a strong regularizer, which is especially beneficial in non-IID data settings common in federated learning.

A class of server-side optimization algorithms (e.g., FedAdam, FedYogi) that use adaptive moment estimation to update the global model. Instead of simple averaging, they apply adaptive, per-parameter learning rates to the aggregated client updates.

• Benefit for Heterogeneity: Provides more stable and faster convergence than FedAvg when client updates are highly varied due to different data distributions and compute capabilities.
• Mechanism: The server maintains its own optimizer state (like momentum vectors) to adjust the aggregation process.

An orchestration strategy that dynamically assigns training tasks to edge clients based on their real-time available computational power, memory, and energy constraints. It queries an on-device resource monitor before selection.

• Goal: Maximize system-wide throughput and completion rates while respecting device limits.
• Implementation: Often integrated with a federated device registry that tracks both static profiles (device type) and dynamic states (battery level, CPU load).