Dynamic Width Networks

The core mechanism of a dynamic width network is its adaptive layer width, where the number of neurons (channels) in a layer can be dynamically scaled. This is often controlled by a width multiplier hyperparameter. For federated learning, the server maintains a full-width model, while each client receives a sub-model with widths scaled according to its compute and memory profile.

• Example: A convolutional layer with 64 filters in the master model might be scaled to 32 filters for a low-tier smartphone and 48 filters for a high-tier tablet.
• Implementation: This is typically achieved through channel masking or structured pruning applied before distributing the model to a client.

All sub-models share weights from the larger master model. When a layer is scaled down, a subset of the master model's neurons and their corresponding weights are selected for the client. This weight sharing is critical for aggregation, as updates from different sub-models must be meaningfully combined.

• Subnet Sampling: In each federated round, a client's specific sub-model configuration (its width configuration) is sampled. This can be random or based on the client's profiled capabilities.
• Aggregation Challenge: The server must correctly map and aggregate the gradient updates from variously sized layers back into the full-width master model's parameter tensor.

A full-capacity master model resides on the federation server. This model represents the maximum architectural width and serves as the parameter repository. It is never directly trained but is updated by aggregating gradients or weights from the heterogeneous client sub-models.

• Role: Acts as the single source of truth for model parameters.
• Aggregation Logic: The server runs specialized aggregation algorithms that can handle partial updates from thinner sub-models, often involving zero-padding or selective parameter averaging to align updates with the master model's structure.

Before local training begins, the client device receives the master model's weights and a width configuration. A lightweight runtime then instantiates the sub-model by extracting the corresponding weights and constructing the thinner network. This process minimizes download size and ensures the model fits the device's RAM constraints.

• Efficiency: Only the necessary parameters are activated for the forward and backward passes, reducing FLOPs and memory footprint.
• Dynamic Adjustment: The width can potentially change per training round based on the device's current resource availability, as reported by an on-device resource monitor.

This architecture is specifically designed to tackle device heterogeneity in cross-device federated learning. It allows participation from a vast pool of devices with varying capabilities—from smart sensors to powerful phones—without excluding the weakest devices or over-provisioning for the strongest.

• Inclusivity: Enables stratified client sampling across capability tiers.
• System Efficiency: Prevents stragglers (slow devices) from dominating round completion time, as they can train a smaller, faster sub-model.
• Privacy Preservation: Maintains the core federated learning benefit of data decentralization while adapting to hardware limits.

It's important to distinguish dynamic width from other adaptive architectures:

• Static Width Networks: Have a fixed architecture, causing issues in federated settings (e.g., out-of-memory errors on low-end devices).
• Dynamic Depth Networks (e.g., Early-Exit Networks): Adapt by changing the number of layers executed. A device may exit early at a shallower layer. This reduces computation but creates aggregation complexity for layers not used by all clients.
• Dynamic Width: Changes the size of each layer. This often provides a more granular and hardware-aligned control over compute/memory usage while maintaining a consistent layer structure across clients, simplifying aggregation.

This is the primary use case. A server maintains a large, high-capacity master model. For each training round, it generates client-specific sub-models by dynamically pruning neurons or channels from this master model based on each device's capability profile (e.g., available RAM, CPU/GPU specs).

• High-end smartphone: Receives a 90% width model, performing extensive local training.
• Low-power IoT sensor: Receives a 30% width model, enabling feasible participation.

All client updates are aggregated at the server to improve the shared master model, ensuring learning from the entire data distribution without excluding weaker devices.

Dynamic width networks enable graceful degradation for inference. A single trained model can be scaled down at deployment time to match the inference latency or power budget of a target device.

• Real-time video analysis on a drone: Uses a wider, more accurate sub-model when battery is high.
• Battery-saver mode: Automatically switches to a narrower, faster sub-model.

This eliminates the need to train and maintain multiple discrete model variants for different device tiers, simplifying the MLOps pipeline and ensuring consistent model behavior across the fleet.

In federated learning, straggler devices with slow compute can delay entire training rounds. Dynamic width networks mitigate this via resource-aware scheduling.

The orchestrator can assign narrower sub-models to slower devices, allowing them to complete their local stochastic gradient descent steps within a synchronized time window. Conversely, faster devices receive wider sub-models, fully utilizing their capacity.

This approach reduces round completion time and prevents the system from being bottlenecked by the slowest participant, a key concern for production federated systems.

Traditional federated learning often has a minimum compute threshold, excluding microcontrollers (MCUs) and TinyML devices. Dynamic width networks can create extremely narrow sub-models (e.g., < 50KB in size) that fit within the SRAM of an ARM Cortex-M series processor.

This unlocks federated learning for:

• Wearable health sensors collecting private biometric data.
• Industrial vibration sensors on motors for predictive maintenance.
• Agricultural soil monitors in remote fields.

By including these devices, the global model learns from data distributions otherwise inaccessible, improving robustness and edge-case performance.

Within an edge computing cluster (e.g., a smart factory with multiple gateways), dynamic width networks facilitate load balancing. A central edge server can distribute inference tasks for a single model across multiple nodes by assigning different width sub-models based on real-time node utilization.

• Underloaded node: Handles a wider sub-model for higher-accuracy tasks.
• Overloaded node: Temporarily shifts to processing requests with a narrower, lighter sub-model.

This provides elastic inference capacity, improving total throughput and system resilience without requiring multiple dedicated models.

Dynamic width networks serve as a powerful substrate for Neural Architecture Search (NAS) and pruning in federated contexts. The performance of different sub-models (widths) trained on real, distributed client data provides a direct signal for architecture optimization.

• The server can analyze which neurons or channels are most frequently activated or contribute most to updates across diverse clients, informing global structured pruning decisions.
• This data-driven approach results in a master model whose architecture is inherently efficient and well-suited for the heterogeneous federated environment, moving beyond pruning based solely on centralized datasets.

A method that splits a neural network into segments, offloading computationally intensive layers to a server or edge node while keeping simpler layers on a resource-constrained device. This enables federated training on devices that cannot hold the full model.

• Key Mechanism: The partition point is determined dynamically based on the client's current available memory and compute.
• Use Case: Allows a smart sensor with limited RAM to train only the first few layers of a vision model, while a nearby gateway handles the deeper layers.

A variant of the core FedAvg algorithm designed to aggregate model updates from clients with vastly different computational capabilities.

• Core Innovation: Moves beyond simple averaging by weighting client contributions based on their local data volume, compute power, or the number of training steps they performed.
• Purpose: Prevents slower, low-capability devices from being drowned out by updates from more powerful clients, leading to a more balanced and fair global model.

A system design paradigm where the global model architecture, training workload, and client participation rules can dynamically scale to match the collective, varying resources of the available client pool.

• Dynamic Width Networks are a primary enabler of this elasticity.
• System Behavior: In a round with many high-capability devices, the system might activate a larger model width. When only constrained devices are available, it automatically scales down to a thinner, more efficient sub-network.

A technique where a random subset of neurons (or entire layers) is deactivated for individual clients during a training round.

• Creates Sub-Models: Each client effectively trains a smaller, unique sub-network, drastically reducing its computational and memory load.
• Dual Benefit: Besides handling heterogeneity, this acts as a strong regularizer, often improving the generalization of the final aggregated global model by combining many different sub-model perspectives.

A communication protocol where the server aggregates client model updates as soon as they are received, without waiting for a synchronized round to finish.

• Solves Straggler Problem: Accommodates clients with highly variable training times due to differences in compute, data size, or connectivity.
• Challenge: Requires sophisticated server-side logic to handle stale updates from slower devices and maintain convergence stability, often using techniques like update weighting based on staleness.

A technique where a global neural network model is pruned to a different sparsity level for each federated client based on its specific compute and memory profile before local training commences.

• Personalized Compression: A high-end phone might receive a model pruned to 80% sparsity, while a microcontroller receives a model at 95% sparsity.
• Relation to Dynamic Width: This is a form of unstructured sparsity, whereas dynamic width networks typically employ structured sparsity (removing entire neurons/filters). Both aim to match the model footprint to device capabilities.