Federated Learning

The fundamental tenet of federated learning is that raw training data never leaves its source device or silo. Instead of a central data lake, the model travels to the data. This means:

• Data Residency: Training occurs where the data is generated (e.g., on a smartphone, hospital server, or factory sensor).
• No Data Centralization: The central coordinating server only ever receives abstract model updates, never raw data samples.
• Inherent Privacy: This architecture provides a strong first line of defense against data breaches at a central point, as sensitive information is never aggregated.

Training proceeds in synchronized rounds of local computation followed by secure aggregation. The canonical algorithm, Federated Averaging (FedAvg), defines this process:

1. Server Broadcast: The central server sends the current global model to a selected subset of clients.
2. Local Training: Each client performs multiple steps of Stochastic Gradient Descent (SGD) on its local dataset.
3. Update Transmission: Clients send only their updated model parameters (or gradients) back to the server.
4. Secure Aggregation: The server computes a weighted average (e.g., by dataset size) of the client updates to form a new global model. This cycle repeats until the model converges, learning from distributed data without ever inspecting it.

A defining challenge of federated learning is that client data is typically Non-Independent and Identically Distributed (Non-IID). This is a core characteristic, not a bug. Data distributions vary significantly across clients due to user behavior, geography, or institutional differences.

• Example: Smartphone keyboard models encounter different vocabulary and emoji use per user.
• Challenge: A single global model may perform poorly for all if data is too heterogeneous. This drives research into personalized federated learning and robust aggregation algorithms.
• Implication: Algorithms must be designed to converge and perform well despite this fundamental distribution skew.

Federated networks, especially in cross-device scenarios, involve a massive number of potential clients (millions of devices), but only a fraction participate in any given training round.

• Client Selection: The server must strategically select clients per round based on availability, connectivity, and resource constraints.
• Systems Challenge: This requires managing stragglers (slow devices) and dropouts (devices that disconnect mid-round).
• Scalability: Protocols must be communication-efficient and asynchronous to handle this scale. Not every device needs to contribute for the global model to improve, embodying a 'wisdom of the crowd' approach.

Unlike centralized training where computation is the bottleneck, in federated learning, network communication between clients and server is often the limiting resource. This shapes algorithm design:

• Communication Compression: Techniques like quantization, sparsification (sending only the largest gradient values), and subsampling are critical to reduce payload size.
• Local Computation: Performing more local SGD steps per round reduces the total number of communication rounds needed for convergence.
• Efficiency Metric: Research focuses on minimizing total communication cost (rounds × bits per round) to achieve target model accuracy.

While data decentralization provides a baseline privacy benefit, federated learning is not inherently perfectly private. Model updates can leak information. Therefore, the paradigm is designed to be compatible with and strengthened by formal privacy techniques:

• Differential Privacy: Adding calibrated noise to client updates before aggregation provides a rigorous, mathematical privacy guarantee bounded by a privacy budget (ε).
• Secure Aggregation: Cryptographic protocols ensure the server only learns the sum of updates from a group of clients, not any individual contribution.
• Trusted Execution Environments (TEEs): Hardware-based secure enclaves (e.g., Intel SGX) can protect the aggregation process. This characteristic makes it a cornerstone of privacy-preserving machine learning.

A rigorous mathematical framework for quantifying and limiting the privacy loss incurred by an individual when their data is included in a statistical analysis or machine learning model. It provides a provable guarantee that the output of an algorithm does not reveal whether any single individual's data was included in the input. In federated learning, differential privacy is often applied by adding calibrated noise to the aggregated model updates before they are used to update the global model.

• Key Parameter: The privacy budget (epsilon, ε) quantifies the maximum allowable privacy loss; a smaller ε provides stronger guarantees.
• Common Mechanism: The Gaussian Mechanism adds noise drawn from a Gaussian distribution to real-valued outputs.

A form of encryption that allows specific types of computations to be performed directly on ciphertext, generating an encrypted result that, when decrypted, matches the result of operations performed on the plaintext. This enables a server to perform computations on encrypted client updates without ever decrypting them, providing an exceptionally strong layer of privacy.

• Fully Homomorphic Encryption (FHE): Supports an unlimited number of arbitrary computations (addition and multiplication) on ciphertexts, enabling general-purpose computation on encrypted data.
• Use in FL: Can be used to encrypt local model updates before they are sent to the aggregating server, though computational overhead is significant.

A cryptographic protocol used in federated learning to compute the sum (or average) of model updates from multiple clients without the central server learning any individual client's contribution. This prevents the server from performing a model inversion attack to infer a client's private data from their update.

• Mechanism: Often relies on Secure Multi-Party Computation principles, where clients mask their updates with secret shares that cancel out only when aggregated across the selected cohort.
• Critical for Cross-Device FL: Essential in scenarios with a massive number of untrusted devices (e.g., smartphones).

The foundational and most widely used algorithm for federated learning. Clients perform multiple local Stochastic Gradient Descent steps on their private data. The server then computes a weighted average of the resulting model updates to form a new global model.

• Core Innovation: Reduces communication rounds by performing more computation locally on each device.
• Challenges: Performance can degrade with highly Non-IID Data across clients, leading to research into personalized and robust variants.
• Weighting: Updates are typically weighted by the number of training examples on each client.

These are the two primary deployment scenarios for federated learning, defined by the nature of the participating clients.

• Cross-Silo Federated Learning: Involves a relatively small number (2-100) of reliable, institutional clients (e.g., hospitals, banks, corporations). Each client has a large, vertically partitioned dataset, and availability is generally high. Focus is on model performance and regulatory compliance.
• Cross-Device Federated Learning: Characterized by a massive number of client devices (millions), such as smartphones or IoT sensors. Each device has a small, non-IID dataset and is highly unreliable (partial participation, intermittent connectivity). Focus is on scalability, communication efficiency, and robustness.

A class of aggregation rules designed for federated learning to tolerate a fraction of malicious or faulty clients (Byzantine nodes) that may send arbitrary or adversarial model updates. These rules aim to produce a reliable global model even under attack, such as Model Poisoning.

• Examples: Krum, Multi-Krum, Median, and Trimmed Mean.
• Principle: Instead of a simple mean, these algorithms select updates based on geometric proximity (Krum) or use robust statistical measures like the median to filter out outliers.
• Trade-off: Increased robustness often comes at the cost of slower convergence under benign conditions.