Federated Learning

Training occurs across a federation of potentially thousands of heterogeneous clients (e.g., smartphones, IoT sensors, edge servers). Key aspects include:

• Client Sampling: Only a subset of available devices participates in each training round, managing resource constraints.
• Statistical Heterogeneity (Non-IID Data): Local data distributions are not independent and identically distributed (Non-IID). A user's phone has data unique to them, creating a major algorithmic challenge.
• Systems Heterogeneity: Devices vary in compute power, network connectivity, and availability, requiring robust protocols to handle stragglers and dropouts.

The global model is refined through synchronized communication rounds, not continuous streaming. The standard Federated Averaging (FedAvg) algorithm follows this cycle:

1. Server Broadcast: The central server sends the current global model to a selected cohort of clients.
2. Local Training: Each client performs several epochs of Stochastic Gradient Descent (SGD) on its local data.
3. Update Transmission: Clients send their updated model weights back to the server.
4. Secure Aggregation: The server aggregates these updates (e.g., by weighted averaging) to form a new global model. Advanced aggregation techniques like FedProx or SCAFFOLD are used to handle system and statistical heterogeneity.

The primary bottleneck is network communication, not computation. Federated learning is designed to be communication-efficient:

• Low-Frequency Updates: Multiple local training steps are performed per communication round, reducing the total number of rounds.
• Compression Techniques: Model updates are compressed via methods like quantization (reducing numerical precision) and sparsification (sending only the largest gradient values) before transmission.
• This focus makes it viable for bandwidth-constrained environments like mobile networks.

The decentralized, semi-trusted environment introduces unique threats that require specialized defenses:

• Byzantine Robustness: Malicious clients may send poisoned updates to sabotage the global model. Aggregation must be robust against such attacks.
• Privacy Attacks: Even model updates can leak information. Techniques like Differential Privacy (DP), which adds calibrated noise to updates, and Secure Multi-Party Computation (SMPC) are layered on for enhanced protection.
• Membership Inference: Adversaries may try to determine if a specific data point was in the training set.

Federated learning manifests in two primary architectural flavors:

• Cross-Device FL: Involves a massive number of intermittent, resource-constrained devices (e.g., millions of smartphones). Characterized by client sampling, high dropout rates, and focus on communication efficiency. Example: Improving next-word prediction on a keyboard app.
• Cross-Silo FL: Involves a small number of reliable, organizational-level clients (e.g., 10 hospitals or 50 banks). Each silo has substantial data and compute. Focus is on collaborative training between institutions where data cannot be pooled due to regulation or competition. Example: Training a diagnostic model across multiple healthcare providers.

A rigorous mathematical framework for quantifying and bounding the privacy loss of individuals when their data is used in statistical analyses. In federated learning, differential privacy is often applied by adding calibrated noise to the model updates (gradients) before they are shared with the central server. This provides a provable guarantee that the aggregated model cannot reveal whether any specific individual's data was used in training.

• Key Mechanism: Adds mathematical noise (e.g., Gaussian, Laplacian) to computations.
• Trade-off: Introduces a privacy-utility trade-off; more noise increases privacy but can reduce model accuracy.

The process of updating or fine-tuning a machine learning model directly on an edge device (e.g., smartphone, IoT sensor) using locally generated data. This is the local training phase of a federated learning round. It emphasizes compute and memory constraints, requiring efficient optimization algorithms and model architectures that can run on limited hardware without external cloud resources.

• Core Challenge: Executing backpropagation and optimizer steps (like SGD) on devices with constrained CPU, RAM, and battery.
• Contrast with Inference: On-device training involves updating model weights, whereas inference is a forward-pass only.

A cryptographic protocol that allows a central server to compute the sum of model updates from multiple clients without being able to inspect any individual client's update. This protects client privacy even from the coordinating server itself. It typically uses multi-party computation (MPC) or homomorphic encryption techniques to mask the updates before transmission.

• Privacy Guarantee: The server learns only the aggregated model update, not the contribution of any single device.
• System Overhead: Introduces communication and computational overhead, which is a key design consideration for federated systems.

A fundamental characteristic and challenge in federated learning where the data distribution across participating devices is not independent and identically distributed (Non-IID). Each device's local dataset may be highly skewed, representing only a subset of classes or a unique feature distribution (e.g., a user's typing habits). This can cause client model drift and destabilize the convergence of the global model.

• Primary Impact: Leads to biased local models that, when averaged, produce a poor global model.
• Mitigation Strategies: Algorithms like FedProx or SCAFFOLD that add regularization or control variates to correct for client drift.

The process of choosing a subset of available devices to participate in a given training round, a critical operational factor in federated learning. Client selection strategies must account for device availability, connectivity, computational resources, and data relevance. Client dropout (devices failing to complete a round) is a common occurrence due to network issues or devices going offline, requiring the aggregation protocol to be robust to partial participation.

• Selection Criteria: Can be random, based on resource availability, or strategic to improve model convergence.
• System Reliability: Aggregation algorithms must be fault-tolerant and not rely on updates from all selected clients.

The two primary organizational paradigms for federated learning, distinguished by the number and type of participants.

• Cross-Silo Federated Learning: Involves a small number (e.g., 2-100) of reliable, organizational-level clients, such as hospitals or financial institutions. Participants have significant computational resources and stable connectivity. The focus is on vertical data partitioning and complex, multi-round training.
• Cross-Device Federated Learning: Involves a massive number (e.g., millions) of unreliable, consumer-level devices like smartphones. Participants have highly constrained resources, unstable connectivity, and must prioritize extreme scalability and privacy. This is the classic use case for on-device personalization.