Memory-Constrained Optimization

A memory-for-compute trade-off technique that dramatically reduces the memory footprint of the backward pass during training. Instead of storing all intermediate activations from the forward pass, only a subset (checkpoints) are saved. During backpropagation, the forward pass is recomputed for non-checkpointed segments. This can reduce memory consumption by up to 5x, at the cost of increased computation time. Essential for training large models on devices with limited RAM.

A strategy where only a subset of a neural network's layers are trained on the client device during a federated round. Common approaches include:

• Freezing early feature extractor layers and only updating later classification layers.
• Cyclic updating, where different layer groups are trained in alternating rounds.
• Importance-based selection, where layers with the highest gradient magnitudes are prioritized. This reduces the memory required to store optimizer states (e.g., momentum) and the computational graph for backpropagation.

A technique inspired by dropout regularization, applied at the system level. In each training round, each client device trains on a randomly selected sub-model created by dropping out a subset of neurons or entire layers from the global architecture. This achieves two goals:

1. Reduces per-client memory/compute load as the active model is smaller.
2. Improves generalization by training an ensemble of sub-models across the federation. The server aggregates updates only for the active parameters in each round, effectively performing sparse aggregation.

Adaptive methods to fit training within fixed memory constraints:

• Dynamic Batching: The local batch size is automatically tuned per device based on its available RAM, preventing out-of-memory (OOM) errors. A high-end phone may use a batch size of 32, while a sensor uses 4.
• Micro-Batching: For very constrained devices, a standard batch is split into micro-batches. Gradients are accumulated across several micro-batches before performing a single optimizer step, simulating a larger effective batch size without the memory overhead of processing all data at once.

The server maintains a single global model but transmits a pruned version tailored to each client's capability profile. Before a training round, the global model is pruned to a target sparsity level (e.g., 50%, 70%) specific to each device's available memory. Clients train this smaller, sparse sub-network. The server then aggregates the sparse updates, often using techniques like weight masking, to update the dense global model. This ensures all devices participate meaningfully, regardless of hardware.

A process that simulates low-precision inference (e.g., 8-bit integers) during the federated training phase. During local client training, forward and backward passes simulate quantization, ensuring the model learns to be robust to the precision loss it will encounter when deployed on edge hardware. This prevents the accuracy drop typically seen when quantizing a model post-training. The server aggregates quantized-aware updates, resulting in a global model that is inherently compatible with the memory-efficient integer arithmetic units common in edge processors.

A strategy where only a subset of a neural network's layers are trained on the client device during a federated round, while other layers remain frozen. This targets updates to the most task-relevant parameters.

• Implementation: The server can specify which layers (e.g., only the final classification head) are trainable for a given round.
• Memory Benefit: Reduces the memory required to store optimizer states (like momentum terms) and gradients, which is proportional to the number of trainable parameters.
• Use Case: Effective for personalization tasks where early, general feature extractors are frozen, and only later, task-specific layers are adapted on-device.

A technique inspired by dropout regularization, applied at the system level. For each training round, a random subset of model parameters or entire layers are 'dropped' (excluded) for each client.

• Memory Reduction: Each client trains a smaller, sparser sub-model, lowering computational and memory load.
• Heterogeneity Benefit: Naturally creates varied model architectures across devices, which can improve generalization when aggregated.
• Protocol: The server must coordinate which parameters are active per round and correctly aggregate the sparse updates.

Adaptively adjusting the local batch size based on a device's available memory. When a full batch does not fit, the batch is split into micro-batches that are processed sequentially with gradient accumulation.

• Core Process: The client's resource monitor determines the maximum viable batch size. Gradients are computed per micro-batch and accumulated to simulate a larger batch update.
• Prevents OOM Errors: Essential for handling variance in client memory, ensuring low-memory devices can still participate without crashing.
• Trade-off: Smaller micro-batches increase iteration time but guarantee completion.

A client-side strategy where the global model is not loaded entirely into memory at once. Instead, it is fetched, instantiated, and trained in stages over multiple communication rounds.

• How it Works: The server and client agree on a model partitioning scheme. The client may train the first few layers in round N, download the next set of layers in round N+1, and so on.
• Benefit: Allows extremely memory-constrained devices (e.g., microcontrollers) to participate in federated learning of models larger than their total RAM.
• Challenge: Requires careful state management and aggregation logic on the server to handle partial model updates.

A memory-for-compute trade-off technique that dramatically reduces the peak memory consumption during neural network training. Instead of storing all intermediate activations needed for backpropagation, it strategically recomputes certain activations on-demand from saved checkpoints.

• Mechanism: Selects a subset of layers to store their output activations (checkpoints). During backpropagation, activations for non-checkpointed layers are recomputed from the nearest checkpoint.
• Impact: Can reduce memory usage by up to 80%, enabling the training of larger models on edge devices, at the cost of increased computation time (typically 20-30% overhead).

A federated optimization strategy where only a designated subset of a neural network's layers are trained on edge devices, while the remaining layers are frozen or updated less frequently.

• Use Case: Critical for fine-tuning large models (e.g., BERT, ResNet) on edge devices. Devices may only update the final classification head or a few task-specific adapters.
• Benefits: Reduces the memory footprint of the optimizer state (e.g., Adam momentum/variance) and the computational load per round. It also decreases the size of the update that must be communicated to the server.

An adaptive technique where the local batch size for on-device training is automatically adjusted per client based on its real-time available memory and compute capacity.

• Implementation: An on-device resource monitor probes available RAM before a training round. The batch size is set to the largest value that can be processed without triggering an out-of-memory error.
• Advantage: Prevents training failures on low-memory devices while allowing better-equipped devices to use larger batches for more stable gradient estimates and hardware efficiency (e.g., better GPU utilization).

A method that creates smaller, more manageable sub-models for each client by randomly dropping out a subset of neurons or entire layers during a training round.

• Process: Each client receives a mask from the server or generates one locally, which zeroes out a random fraction of model weights. The client trains only this sparse sub-model.
• Outcomes: Significantly reduces the active parameter count and memory usage per client. When aggregated across many rounds, the server's global model effectively trains as the full dense network. This also acts as a strong regularizer, improving generalization.

A model compression technique applied client-side, where the global neural network is pruned to a different sparsity level for each federated client based on its specific memory and compute profile.

• Workflow: The server maintains a master dense model. For each training round, it generates a uniquely pruned model copy for a client using a pruning criterion (e.g., magnitude-based) tuned to that client's capability tier.
• Result: Low-tier devices train extremely sparse models (e.g., 90% pruned), while high-tier devices train moderately sparse ones. Aggregation on the server reconstructs a dense, high-quality global model.

A federated training scheme where each client only trains a randomly selected subset of the global model's total parameters in each round.

• Mechanism: Similar to DropConnect, a random mask is applied to the model's weight matrices. The client computes gradients and updates only for the unmasked weights.
• Memory Advantage: The optimizer state (e.g., for SGD with momentum) is only maintained for the active parameters, leading to substantial memory savings. This allows the use of adaptive optimizers like Adam on memory-constrained devices, which is otherwise prohibitive due to Adam's requirement to store two auxiliary variables per parameter.