FedAdam

Unlike standard Federated Averaging (FedAvg) which performs a simple weighted average of client updates, FedAdam applies the Adam optimizer on the server. The server treats the aggregated client update as a pseudo-gradient and uses Adam's adaptive learning rates (maintaining first-moment and second-moment estimates) to update the global model. This provides per-parameter learning rate adjustments based on past update history.

A primary benefit of FedAdam is its robustness to statistical heterogeneity (non-IID data). The adaptive moment estimation helps correct for client drift, where local models diverge due to optimizing on different data distributions. By adjusting update magnitudes parameter-by-parameter, FedAdam can stabilize convergence paths that would oscillate or diverge under simple averaging.

FedAdam is a canonical example within the generalized FedOpt framework. This framework abstracts the server update step as: server_optimizer.apply_gradients(aggregated_client_deltas). FedAdam instantiates this with an Adam optimizer. Other variants in the same family include FedYogi and FedAdagrad, which use different adaptive optimizers on the server.

Introduces key server-side hyperparameters beyond the client learning rate:

• Server learning rate (η): The global step size for the Adam update.
• Adam β₁, β₂: Exponential decay rates for the moment estimates (typically 0.9 and 0.999).
• ε: A small constant for numerical stability. Tuning these is critical for performance, as the server learning rate often needs to be larger than the client learning rate used in FedAvg.

Empirical and theoretical analyses show FedAdam can achieve faster convergence than FedAvg, particularly under high data heterogeneity. The adaptive updates can navigate complex loss landscapes more efficiently. The convergence guarantee typically requires assumptions like bounded client gradients and convexity, but demonstrates improved dependency on data variance terms compared to FedAvg.

FedAdam maintains the same communication pattern as FedAvg—clients send model deltas, server broadcasts the new global model. The added computational cost is minimal and confined to the server, which must maintain and update the moment vectors for all model parameters. This makes it a practical upgrade from FedAvg without increasing client-side burden or communication frequency.

FedAdam replaces the simple weighted averaging step in Federated Averaging (FedAvg) with an adaptive optimizer on the server. The server maintains first-moment (m) and second-moment (v) estimates of the aggregated client gradients. It then applies the Adam update rule:

• Server computes pseudo-gradient: Δ = Σ (weightᵢ * client_deltaᵢ)
• Updates moments: m = β₁*m + (1-β₁)Δ, v = β₂v + (1-β₂)*Δ²
• Applies correction and update: θ = θ - η * (m̂ / (√v̂ + ε)) This adapts the global learning rate per parameter, often leading to faster and more stable convergence than FedAvg, especially with heterogeneous client data.

FedAdam is a specific instantiation of the generalized FedOpt framework. FedOpt abstracts the server aggregation step as an optimization problem, allowing the use of any gradient-based optimizer:

• FedAvg: Server optimizer = SGD
• FedAdam: Server optimizer = Adam
• FedYogi: Server optimizer = Yogi
• FedAdagrad: Server optimizer = Adagrad This framework provides a unified view for analyzing and developing new federated optimization algorithms by decoupling client-side local SGD from the server's update rule.

Tuning FedAdam requires setting server-specific hyperparameters in addition to standard federated learning ones:

• Server Learning Rate (η): The global step size. Often set lower than client LR.
• β₁, β₂: Exponential decay rates for the moment estimates (e.g., 0.9, 0.999).
• ε: A small constant for numerical stability (e.g., 1e-8).
• Client Learning Rate & Local Epochs: Control local optimization depth.
• Momentum vs. Adaptive Methods: FedAdam's β₁ provides momentum-like acceleration, while β₂ provides per-parameter adaptive scaling, distinguishing it from FedAvgM which uses a single momentum scalar.

FedAdam addresses several limitations of vanilla FedAvg:

• Faster Convergence: Adaptive per-parameter learning rates can accelerate progress, especially in early training or with sparse gradients.
• Improved Stability: Automatic scaling of updates can be more robust to varying gradient magnitudes across clients.
• Handling Heterogeneity: Can better navigate the complex loss landscapes arising from non-IID data by adapting the server update direction.
• Reduced Hyperparameter Sensitivity: Less sensitive to the exact choice of server learning rate compared to FedAvg's SGD step.

Deploying FedAdam introduces specific system and algorithmic considerations:

• Increased Server Memory: The server must store two additional moment tensors (m, v) equal in size to the model, doubling server-state memory overhead.
• Communication Cost Unchanged: Client-server communication costs are identical to FedAvg; only the server's aggregation logic changes.
• Client-Server Mismatch: The adaptive update is computed from aggregated gradients, not true gradients, which can bias moment estimates. Techniques like FedAdam with correction adjust for this.
• Convergence Guarantees: Theoretical analysis is more complex than for FedAvg but proofs exist under similar assumptions of bounded gradients and convexity.

FedAdam is part of a family of adaptive federated optimizers:

• FedYogi: Uses a more conservative update for the second moment (v), which can be more stable in non-stationary federated settings.
• FedAdagrad: Adapts learning rates based on the sum of historical squared gradients, often too aggressive decay in deep learning.
• FedAvgM (Momentum): Incorporates a simpler momentum term without per-parameter adaptation.
• Adaptive Federated Optimization (AFO): A broader class exploring how to best leverage adaptive methods on the server while accounting for client sampling and data heterogeneity.

The FedOpt framework is a generalized server-side optimization paradigm for federated learning. Instead of using simple averaging (like FedAvg), it allows the central server to apply adaptive optimization algorithms—such as Adam, Yogi, or Adagrad—to the aggregated client updates. This framework provides a flexible template for designing algorithms that can better handle non-IID data and varying client participation.

• Core Idea: Treat the aggregated client update as a pseudo-gradient and apply an adaptive optimizer on the server.
• Key Benefit: Enables the use of momentum and per-parameter learning rate adjustments at the server level, which can stabilize and accelerate convergence.
• Example: FedAdam and FedYogi are specific instantiations of the FedOpt framework.

FedAvgM (Federated Averaging with Server Momentum) incorporates a momentum term into the server's aggregation step. After computing the weighted average of client updates, the server applies a momentum buffer, similar to traditional SGD with momentum, to update the global model.

• Mechanism: server_state = momentum * server_state + aggregated_update; global_model = global_model - server_learning_rate * server_state.
• Purpose: Momentum helps smooth out the update trajectory, dampening oscillations caused by client heterogeneity and improving convergence speed, especially in settings with partial client participation.
• Relation to FedAdam: FedAvgM can be seen as a precursor or a specific case within the adaptive optimizer family, where the update is adapted using a single, global momentum buffer rather than per-parameter adaptive learning rates.

SCAFFOLD (Stochastic Controlled Averaging for Federated Learning) is an algorithm designed to correct for client drift—the divergence of local client models due to non-IID data. It introduces control variates (correction terms) for both the server and each client to reduce the variance between local and global update directions.

• Key Innovation: Clients compute a difference between their local and global control variates to correct their update before sending it to the server.
• Contrast with FedAdam: While FedAdam adapts the server's update rule, SCAFFOLD modifies the client's local objective and update computation. SCAFFOLD provides stronger theoretical convergence guarantees under heterogeneity but requires storing and communicating additional state (the control variates).

FedProx is a federated optimization algorithm that addresses statistical heterogeneity by adding a proximal term to the local loss function on each client. This term penalizes the local model for deviating too far from the global model received at the start of the round.

• Local Objective: Local Loss + (mu/2) * ||local_model - global_model||^2 where mu is a hyperparameter.
• Effect: The proximal term acts as a regularizer, explicitly limiting client drift and making local updates more consistent. It also allows for variable amounts of local work (e.g., different numbers of local steps) across clients.
• Comparison: FedProx operates by modifying the client-side optimization, whereas FedAdam modifies the server-side optimization. They address similar challenges (non-IID data) from different angles and can be conceptually complementary.

Adaptive optimization algorithms like Adam and Yogi are the foundation upon which FedAdam is built. These optimizers compute individual adaptive learning rates for each model parameter based on estimates of the first moment (the mean of gradients) and the second moment (the uncentered variance of gradients).

• Adam (Adaptive Moment Estimation): Combines momentum (first moment) and RMSProp (second moment) with bias correction. It is known for robust performance in centralized deep learning.
• Yogi: A modification of Adam that provides more robust second-moment estimation, which can be beneficial in the noisy, decentralized setting of federated learning.
• Federated Adaptation: In FedAdam, the server uses these optimizers' update rules, treating the aggregated client update as a pseudo-gradient to compute the moments and adapt the global model's learning rates per parameter.

FedNova (Federated Normalized Averaging) is an aggregation scheme designed to account for heterogeneity in local client workloads. When clients perform different numbers of local epochs or steps, their update magnitudes vary, which can bias the simple weighted average.

• Core Method: FedNova normalizes each client's update by the number of local steps taken before averaging them on the server. This ensures updates are combined fairly, irrespective of local computation effort.
• Problem it Solves: Prevents clients that perform more work from disproportionately dominating the global update direction, leading to more stable convergence.
• Synergy with FedOpt: Normalization in FedNova addresses one source of variance, while adaptive optimization in FedAdam addresses another. They can be combined; for example, the normalized updates from FedNova can serve as the input to the FedAdam optimizer on the server.