Federated Learning

The most defining characteristic of federated learning is that raw training data never leaves its source device or organizational silo. Instead of a central data warehouse, the model travels to the data. This architecture is governed by the principle of data minimization, ensuring the data owner retains physical and legal control. This is critical for compliance with regulations like GDPR and HIPAA, where data locality is a legal requirement. For example, a keyboard prediction model learns from typing patterns directly on a user's phone without sending keystrokes to a cloud server.

Federated learning systems inherently operate on Non-Independent and Identically Distributed (Non-IID) data. Each client's local dataset is generated by its unique usage patterns and environment, creating significant statistical differences across the network.

• Causes: User behavior, geographic location, device type, and time of day all contribute to unique local distributions.
• Challenge: This violates the core IID assumption of traditional stochastic gradient descent, leading to client drift where local models diverge, slowing convergence and harming final accuracy.
• Solution: Algorithms like FedProx and SCAFFOLD are explicitly designed to mitigate the effects of heterogeneity by constraining local updates or using control variates.

Federated learning deployments fall into two primary scales with distinct system characteristics:

• Cross-Device FL: Involves a massive number of resource-constrained, intermittently connected devices (e.g., millions of smartphones). Key traits are partial participation per round, unreliable connectivity, and severe system heterogeneity (varied compute, memory, battery).
• Cross-Silo FL: Involves a small number (e.g., 2-100) of reliable, resource-rich organizational entities (e.g., hospitals, banks). Key traits are full participation potential, higher reliability, and a focus on vertical federated learning where parties hold different features for the same entities.

The algorithmic and systems design differs drastically between these two paradigms.

In federated learning, communication is often the primary bottleneck, not computation. Transmitting full model updates from millions of devices to a central server is prohibitively expensive. Therefore, FL research heavily focuses on communication compression techniques:

• Model Compression: Techniques like quantization (reducing numerical precision of updates) and sparsification (sending only the largest gradient values).
• Local Steps: Performing multiple steps of Local SGD on the client reduces the frequency of communication rounds.
• Server-Side Techniques: Using adaptive server optimizers like FedAdam that can converge effectively with fewer or compressed client updates.

The goal is to achieve high model accuracy with a minimal number of communicated bits.

While raw data stays local, shared model updates can still leak information. A core characteristic of robust FL is the use of cryptographic and algorithmic techniques to provide multi-layered privacy guarantees during aggregation.

• Secure Aggregation: A cryptographic protocol that allows the server to compute the sum of client updates without being able to inspect any individual contribution.
• Differential Privacy (DP): Adds carefully calibrated noise to client updates before they are sent, providing a mathematically rigorous bound on privacy loss. This creates a direct privacy-accuracy trade-off.
• Homomorphic Encryption: Allows the server to perform computations on encrypted model updates, though it is computationally intensive.

These techniques defend against gradient leakage and membership inference attacks.

The federated environment is inherently unreliable and potentially adversarial. FL systems must be designed for Byzantine Robustness and fault tolerance.

• Partial Client Participation: In any given communication round, only a subset of clients may be available due to connectivity or power constraints. The system must function correctly with this stochastic availability.
• Byzantine Clients: Malicious participants may send poisoned updates to perform model poisoning or backdoor attacks. Robust aggregation rules (e.g., median-based, trimmed mean) are used to filter out outliers.
• Straggler Mitigation: Devices with slow compute can delay rounds. Techniques like asynchronous aggregation or deadline-based updates are used to maintain system throughput.

Federated Averaging (FedAvg) is the foundational algorithm for model aggregation. The central server computes a weighted average of client model updates to form a new global model. Its core steps are:

• Server broadcasts the current global model to a subset of clients.
• Each client performs Local SGD on its private data.
• Clients send their updated model weights back to the server.
• The server aggregates updates, typically weighting them by the number of local training samples. FedAvg's simplicity makes it the baseline, but it struggles with Non-IID Data and Client Drift.

Differential Privacy (DP) is a rigorous mathematical framework for quantifying and bounding privacy loss. In FL, it ensures a client's participation in training does not reveal its specific data. Implementation involves:

• Adding calibrated noise (e.g., Gaussian) to client updates before aggregation.
• Clipping updates to bound their sensitivity. This creates a fundamental Privacy-Accuracy Trade-off; stronger privacy guarantees often reduce final model accuracy. DP is a cornerstone of regulatory-compliant FL systems.

Secure Aggregation is a cryptographic protocol that allows a server to compute the sum of client model updates without inspecting any individual contribution. It protects against a curious central server. Key properties include:

• The server learns only the aggregated model update, not individual client vectors.
• It often uses Secure Multi-Party Computation (SMPC) or masking techniques.
• It is complementary to Differential Privacy; DP protects the output, while Secure Aggregation protects the inputs during transmission and aggregation.

Statistical Heterogeneity, manifesting as Non-IID Data across clients, is the defining characteristic of real-world FL. Client data distributions vary in:

• Feature distribution (covariate shift).
• Label distribution (prior probability shift).
• Same label, different features (concept shift). This heterogeneity causes Client Drift, where local models diverge, slowing convergence and harming global model performance. Algorithms like FedProx and SCAFFOLD are explicitly designed to mitigate this challenge.

Personalization refers to techniques that adapt a global FL model to individual client data distributions. Since a single global model may perform poorly on heterogeneous clients, personalization strategies include:

• Training local Adapter Layers on top of a frozen global model.
• Using Low-Rank Adaptation (LoRA) for efficient on-device fine-tuning.
• Learning client-specific model parameters or performing meta-learning. The goal is to balance the shared knowledge of the global model with the specificity needed for optimal local performance.

Byzantine Robustness is the property of an FL aggregation algorithm to tolerate malicious or faulty clients. These Byzantine clients may send arbitrary updates to perform Model Poisoning or Backdoor Attacks. Robust aggregation techniques include:

• Median-based or trimmed mean aggregation, discarding extreme updates.
• Krum, which selects the update most similar to its peers.
• Redundancy-based schemes requiring multiple honest clients. Ensuring Byzantine robustness is critical for FL security in open or adversarial environments.