Glossary
Federated Optimization Techniques
Federated Optimization Techniques
Terms related to specialized optimization methods designed for the federated learning setting. Target: ML Engineers & Researchers.
Federated Averaging (FedAvg)
Federated Averaging (FedAvg) is the foundational iterative optimization algorithm for federated learning, where a central server aggregates locally computed model updates from a subset of clients by taking a weighted average to produce a new global model.
FedProx
FedProx is a federated optimization algorithm that modifies the local client objective by adding a proximal term to constrain local updates, thereby improving convergence and stability when dealing with statistical and systems heterogeneity across clients.
SCAFFOLD
SCAFFOLD (Stochastic Controlled Averaging for Federated Learning) is an optimization algorithm that uses control variates to correct for client drift caused by data heterogeneity, leading to significantly faster convergence compared to standard Federated Averaging.
FedOpt
FedOpt is a framework for federated optimization that generalizes the server-side update step of Federated Averaging, allowing the use of adaptive optimizers like Adam, Yogi, or Adagrad on the global model instead of simple averaging.
FedAdam
FedAdam is a federated optimization algorithm within the FedOpt framework that applies the Adam adaptive optimizer to the server's aggregation of client updates, improving performance on non-convex problems.
FedYogi
FedYogi is a federated adaptive optimization algorithm that adapts the Yogi optimizer for server-side aggregation, offering more stable convergence than FedAdam, particularly in the presence of noisy client gradients.
FedAdagrad
FedAdagrad is a federated optimization algorithm that applies the Adagrad adaptive learning rate method during the server's model aggregation step, assigning smaller updates to frequently occurring features in the global model.
Local Stochastic Gradient Descent (Local SGD)
Local Stochastic Gradient Descent (Local SGD) is the core client-side training procedure in federated learning, where each selected device performs multiple iterations of SGD on its local dataset before communicating its model update to the server.
Client Drift
Client drift is a phenomenon in federated learning where local client models diverge from the global objective due to performing multiple steps of optimization on statistically heterogeneous (non-IID) local data, hindering global convergence.
Adaptive Federated Optimization
Adaptive Federated Optimization refers to a class of federated learning algorithms that incorporate adaptive learning rate methods, such as Adam or Adagrad, either on the server, client, or both sides to improve convergence speed and stability.
Gradient Compression
Gradient compression is a communication-efficient technique in federated learning that reduces the size of model updates sent from clients to the server using methods like sparsification, quantization, or low-rank approximations.
Quantized Gradient Communication
Quantized Gradient Communication is a compression technique for federated learning where high-precision gradient values are mapped to a lower-bit representation before transmission to drastically reduce communication bandwidth.
Top-k Sparsification
Top-k Sparsification is a gradient compression method where only the k largest magnitude values (by absolute value) in a gradient tensor are transmitted, while the rest are set to zero, significantly reducing communication cost.
Error Feedback
Error Feedback is a mechanism used in conjunction with gradient compression techniques that accumulates the compression error locally and adds it back to the next gradient computation, preserving convergence guarantees.
Probabilistic Client Participation
Probabilistic Client Participation is a client sampling strategy in federated learning where devices are selected for each training round based on a probability distribution, often weighted by data quantity or system readiness.
Active Client Selection
Active Client Selection is a strategic approach in federated learning where the server chooses participants for a training round based on criteria designed to improve learning efficiency, such as data quality, resource availability, or update significance.
Asynchronous Federated Optimization
Asynchronous Federated Optimization is a training paradigm where the central server updates the global model immediately upon receiving an update from any client, without waiting for a synchronized round, improving efficiency in heterogeneous environments.
FedAsync
FedAsync is an asynchronous federated learning algorithm where the server aggregates a client's stale update using a mixing hyperparameter that decays with the update's age, mitigating the negative effects of system asynchrony.
Personalized Learning Rates
Personalized Learning Rates in federated learning involve assigning different learning rate schedules or values to individual clients based on their local data distribution or computational characteristics to improve personalized model performance.
Heterogeneous Client Optimization
Heterogeneous Client Optimization refers to federated learning algorithms and strategies specifically designed to handle variations in client data distributions (statistical heterogeneity), hardware capabilities, and network connectivity.
Federated Variance Reduction
Federated Variance Reduction encompasses techniques adapted from classical optimization, like SVRG or SAGA, to reduce the variance of stochastic gradients in the federated setting, accelerating convergence under data heterogeneity.
Federated SVRG
Federated SVRG is an adaptation of the Stochastic Variance Reduced Gradient (SVRG) algorithm for federated learning, which uses control variates to reduce gradient variance and improve convergence speed in heterogeneous environments.
Federated Mirror Descent
Federated Mirror Descent is a generalization of federated optimization that performs gradient updates in a dual space defined by a Bregman divergence, offering a unified framework for handling constraints and non-Euclidean geometry.
Federated Second-Order Optimization
Federated Second-Order Optimization refers to methods that incorporate curvature information (via the Hessian or its approximations) into the federated learning update to achieve faster convergence, though at increased computational and communication cost.
Federated Natural Gradient
Federated Natural Gradient is an optimization method that preconditions client gradients using (an approximation of) the Fisher information matrix, providing an update direction that accounts for the geometry of the model's probability distribution.
Federated Meta-Learning
Federated Meta-Learning applies meta-learning principles, such as Model-Agnostic Meta-Learning (MAML), within a federated framework to learn a global model initialization that can be rapidly adapted to new clients with minimal local data.
Federated Hyperparameter Optimization
Federated Hyperparameter Optimization is the process of tuning model and algorithm hyperparameters (e.g., learning rates, local epochs) in a federated learning system without centralizing client data, often using Bayesian optimization or population-based methods.
Federated Multi-Task Learning
Federated Multi-Task Learning is a paradigm where a federated system learns a set of related but distinct tasks simultaneously across clients, sharing representations to improve individual task performance while respecting data locality.
Federated Reinforcement Learning Optimization
Federated Reinforcement Learning Optimization involves training reinforcement learning policies in a federated manner, where multiple agents interact with different environments and share policy updates to learn a robust global policy.
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us