Dynamic Batching

The technique relies on a lightweight on-device resource monitor that continuously tracks key metrics:

• Available RAM to prevent out-of-memory (OOM) errors.
• CPU/GPU/NPU utilization to avoid system lag.
• Battery level and thermal state to preserve user experience and device integrity.
• Network bandwidth for subsequent update transmission. This live telemetry is the primary input for the batching decision algorithm, allowing the system to react to changing conditions like a user starting a power-intensive app.

Based on the resource monitor's input, a heuristic or learned policy calculates the maximum viable batch size. This is not a simple linear scaling. Key considerations include:

• Memory-bound calculation: Estimating the peak memory footprint of a forward/backward pass for a given batch size.
• Compute-throughput trade-off: Larger batches improve hardware utilization (e.g., GPU parallelism) but increase per-iteration latency and memory pressure.
• Energy efficiency: Larger batches may complete the target local epoch count faster, potentially saving energy versus many small, inefficient batches.

Dynamic batching interacts directly with compute-aware selection strategies. The federation server can query devices for their current effective batch size as a proxy for readiness. A device in a low-power state may report a batch size of 1 or 2, signaling it should be deprioritized for the current training round. This allows the orchestrator to form a cohort of clients that can train with reasonable efficiency, improving round completion times and system predictability.

To ensure training stability, dynamic batching implements safeguards against excessive variance in client updates:

• Minimum batch size: Enforces a floor (e.g., 4) to maintain a meaningful gradient signal and prevent extreme noise.
• Gradient accumulation: On very constrained devices, the system can simulate a larger batch by accumulating gradients over several micro-batches before applying an update, preserving convergence properties.
• Server-side normalization: During federated averaging, the server can weight updates by the actual batch size used, mitigating the bias introduced by clients with vastly different batch sizes.

Dynamic batching is often used in conjunction with capability-based pruning and federated quantization-aware training (FQAT). The workflow is synergistic:

1. A device receives a global model already pruned or quantized to match its static capability profile.
2. Within that model footprint, dynamic batching then adjusts the runtime batch size based on dynamic resource availability. This two-tiered approach—static model adaptation plus dynamic batching—provides fine-grained control over the computational load.

In asynchronous federated update settings, dynamic batching is critical for managing stragglers. A slow device with a small batch size will not block the server. The protocol handles:

• Staleness weighting: Older updates from slow clients may be discounted.
• Continuous local training: Devices can proceed at their own pace, recalculating batch size each epoch if resources change, without being locked to a synchronous round clock. This maximizes device participation but requires more sophisticated aggregation logic on the server.

A federated learning system trains a vision model for a photo app. Dynamic batching adjusts the local batch size per device:

• A flagship phone with 8GB RAM uses a batch size of 32.
• A mid-range phone with 4GB RAM uses a batch size of 16.
• An older phone with 2GB RAM uses a batch size of 8 or 4. This prevents out-of-memory (OOM) errors and thermal throttling, ensuring all devices can participate without crashing. The server receives updates from a more diverse client pool, improving model robustness.

A network of industrial sensors monitors equipment vibration. These sensors have severely constrained memory (e.g., 512KB RAM).

• Dynamic batching reduces the batch size to 1 (online learning) or a small mini-batch of 2-4.
• The training loop uses gradient accumulation to simulate a larger batch over multiple steps before applying updates.
• This allows the tinyML model to train directly on the sensor, learning local fault patterns without exceeding memory limits or draining the battery with constant communication.

In a fleet of delivery vehicles, onboard computers train a route optimization model. Network connectivity is sporadic.

• When a strong Wi-Fi connection is predicted at a depot, the system increases the batch size to perform more local computation, creating a higher-quality update to send in a single transmission.
• When on a cellular link with poor bandwidth, it reduces the batch size to train faster and send smaller, more frequent updates, preventing a single large transmission from failing. This connectivity-aware batching maximizes useful learning per byte transmitted.

A federated system personalizes a health monitoring model on smartwatches. Battery and thermal limits are critical.

• At 80% battery, the watch uses a standard dynamic batch size.
• When battery falls below 20%, the batch size is minimized to 1 to reduce CPU usage.
• If the device's internal temperature sensor indicates heating, batch size is reduced or training is paused entirely. This battery-aware dynamic batching prioritizes user experience and device safety while still enabling incremental learning.

For highly heterogeneous clients, the federation server plays an active role.

1. Client Profiling: The server maintains a registry of device capabilities (CPU cores, RAM).
2. Initial Assignment: At round start, the server suggests an initial batch size based on profile.
3. Runtime Feedback: The client's on-device resource monitor reports memory pressure. If high, the client self-adjusts the batch size downward.
4. Aggregation Adjustment: The server uses techniques like weighted averaging or HeteroFA to account for the variable amount of local work (influenced by batch size) when aggregating updates. This creates a closed-loop system for resource management.

Dynamic batching is often combined with other heterogeneity techniques.

• Step 1: A global model is pruned via capability-based pruning; a lighter sub-model is sent to a weak client.
• Step 2: Dynamic batching is applied to this already-small model to find the optimal batch size for the client's current available RAM.
• Step 3: The client may also use gradient checkpointing to trade compute for memory, allowing a slightly larger batch. This multi-layered adaptation ensures the local training task is precisely tailored to the device's instantaneous capacity, maximizing participation and system throughput.

A federated learning orchestration strategy that dynamically assigns training tasks to edge clients based on their real-time available computational power, memory, and energy constraints. Unlike static schedules, it continuously evaluates device state to:

• Prioritize clients with surplus CPU/GPU cycles.
• Avoid overloading devices nearing memory limits.
• Defer tasks for devices with low battery. This ensures efficient round completion and protects user experience.

The systematic process of measuring and cataloging the static and dynamic resources of enrolled edge devices. This creates a registry used by the server for intelligent orchestration. Profiled metrics include:

• Static: CPU architecture, total RAM, NPU support.
• Dynamic: Available memory, battery percentage, network bandwidth, thermal status. This foundational data enables techniques like dynamic batching and compute-aware selection.

A suite of techniques that modify the federated training process to operate within the strict RAM limitations of edge devices. These are often used in conjunction with dynamic batching and include:

• Gradient Checkpointing: Trading compute for memory by re-calculating activations during the backward pass.
• Selective Layer Updating: Only computing gradients for a subset of model layers per round.
• Capability-Based Pruning: Transmitting a sparser sub-model to clients based on their memory profile.

A federated learning protocol that allows clients to perform different numbers of local SGD steps based on their available resources, rather than enforcing a fixed epoch count. A device with high compute might perform 10 local epochs, while a constrained device may only manage 2. The server must then account for this heterogeneity in local computation when aggregating updates, often using weighting schemes based on steps or data processed.

A communication protocol where the server aggregates client model updates as soon as they are received, without waiting for a synchronized round to finish. This accommodates clients with highly variable training times caused by dynamic batching and other heterogeneity factors. Benefits include:

• Higher device participation rates.
• Reduced idle time for fast clients.
• Challenges include managing update staleness and designing stable aggregation for non-iid updates.

A technique that randomly drops out a subset of neurons or entire layers for individual clients during a training round. This creates smaller, client-specific sub-models that reduce computation and memory load—acting as a form of architectural dynamic batching. Benefits include:

• Lower per-client resource requirements.
• A form of regularization that can improve generalization.
• Enables participation from very low-capacity devices. The server aggregates updates only for the active parameters in each round.