FedOpt

FedOpt fundamentally separates the local training procedure from the global aggregation step. Clients perform standard stochastic gradient descent (SGD) on their private data, while the server applies a distinct server optimizer to the pseudo-gradient formed by the difference between the aggregated client updates and the current global model. This decoupling allows the server to leverage momentum, adaptive learning rates, and other sophisticated optimization techniques without modifying client-side code.

The framework's key innovation is treating the aggregated client update as a gradient estimate for a server-level optimizer. Supported server optimizers include:

• FedOpt-Adam: Applies the Adam optimizer with its adaptive per-parameter learning rates and momentum to the server update, accelerating convergence on sparse or noisy gradients.
• FedOpt-Yogi: A variant of Adam with improved convergence guarantees under adversarial or highly heterogeneous client updates.
• FedOpt-SGD with Momentum: A simpler alternative that adds a momentum term to the server update for smoother global model trajectories.

FedOpt constructs a pseudo-gradient on the server by computing the difference between the current global model weights and the weighted average of received client model updates. This pseudo-gradient is then fed into the server optimizer as if it were a standard stochastic gradient. The process is mathematically expressed as: Δ_t = (1/η) * (w_t - w_avg), where w_t is the current global model, w_avg is the aggregated client update, and η is the client learning rate. This formulation enables the server optimizer to treat the federated update as a standard optimization step.

In standard FedAvg, heterogeneous local data distributions cause client drift, where local models diverge from the global optimum during multiple local epochs. FedOpt mitigates this by applying server-side momentum that smooths the update trajectory. The server optimizer's momentum term acts as a low-pass filter, dampening the variance introduced by divergent local updates. This is particularly effective when combined with FedOpt-Adam, whose adaptive learning rates automatically scale down updates from clients with noisy or outlier gradients.

FedOpt reduces the total number of communication rounds required for convergence compared to vanilla FedAvg. By applying adaptive server optimization, the global model takes more informed steps toward the optimum per round, requiring fewer synchronization cycles. Empirical studies demonstrate that FedOpt-Adam can achieve target accuracy in 30-50% fewer communication rounds than FedAvg on non-IID data partitions, directly translating to reduced bandwidth costs and faster time-to-deployment in cross-silo healthcare networks.

FedOpt is designed as a drop-in replacement for the server-side aggregation logic in existing federated learning systems. It is fully compatible with:

• Secure Aggregation (SecAgg): The pseudo-gradient can be computed over encrypted client updates without exposing individual contributions.
• Differential Privacy: Noise can be injected into the pseudo-gradient before server optimization to provide formal privacy guarantees.
• Client Sampling: FedOpt works with random subsets of clients per round, maintaining convergence properties under partial participation.
• Heterogeneous Local Epochs: Clients can perform varying numbers of local SGD steps without breaking the server optimization framework.