Communication Cost

The total number of parameters in the neural network directly dictates the baseline cost per communication round. Transmitting a full model update for a model with millions or billions of parameters (e.g., ResNet-50 has ~25 million) consumes significant bandwidth. Techniques like gradient sparsification (sending only the top-k% of values) and structured updates reduce this dimensionality.

The number of clients selected per training round (K out of N total clients) scales communication linearly. More participants increase total upstream bandwidth but can improve statistical efficiency. Adaptive client selection strategies aim to minimize rounds by prioritizing clients with:

• High-quality data or large local datasets
• Good network connectivity
• Sufficient computational resources to avoid being stragglers

The number of local training epochs (E) performed on a client before communicating an update creates a fundamental trade-off. More local computation reduces communication rounds but can cause client drift, where local models diverge from the global objective, ultimately requiring more rounds to converge. Finding the optimal E is critical for total cost.

Applying lossy compression to model updates is a primary method for cost reduction.

• Quantization: Reducing the numerical precision of gradients from 32-bit floats to 8-bit integers or lower.
• Sparsification: Only sending gradients whose magnitude exceeds a threshold.
• Subsampling: Transmitting a random subset of parameters. These methods introduce noise, which must be managed to ensure convergence.

The physical and logical communication structure imposes overhead.

• Client-Server (Star): Standard in cross-device FL; server is a single point of congestion.
• Peer-to-Peer (Decentralized): Eliminates the server but requires complex coordination.
• Protocol Overhead: Headers, encryption for secure aggregation, and handshakes add fixed cost per message, which is significant for small, frequent updates.

Variations in client hardware, network speed, and availability create stragglers—slow clients that delay the aggregation of each round. Strategies to mitigate this increase cost:

• Timeout-based aggregation: Proceeds without slow clients, wasting their computation.
• Asynchronous updates: Reduces idle time but can harm convergence stability. Managing stragglers often involves a direct trade-off between total training time and total bytes transmitted.

A subfield focused on techniques to minimize the bandwidth and frequency of data exchange between clients and the server. Core methods include:

• Model Compression: Techniques like quantization and pruning to reduce update size.
• Sparsification: Transmitting only the most significant model updates (e.g., top-k gradients).
• Local Training: Increasing the number of local epochs between communications to amortize cost. The goal is to maintain model performance while drastically reducing the communication overhead, which is often the dominant cost factor.

A set of techniques used to reduce the memory footprint and computational requirements of a neural network, directly impacting communication cost. Key methods are:

• Quantization: Reducing the numerical precision of model weights (e.g., from 32-bit floats to 8-bit integers).
• Pruning: Removing redundant or less important weights or neurons from the model.
• Knowledge Distillation: Training a smaller "student" model to mimic a larger "teacher" model. In federated learning, compressed models result in smaller update payloads, lowering per-round communication cost.

The variation in computational capability, memory, network connectivity, and availability across client devices. This directly impacts communication efficiency:

• Stragglers: Slow clients delay the aggregation of updates for a synchronous round.
• Unreliable Networks: Clients with poor or intermittent connectivity may drop out, wasting bandwidth.
• Adaptive Protocols: Systems must handle clients that can only transmit partial updates or connect infrequently. Managing heterogeneity is essential for minimizing wasted communication and ensuring efficient training.

Measures how quickly the global model's loss decreases or its accuracy increases per communication round. It is intrinsically linked to communication cost:

• A slower convergence rate requires more rounds to achieve target accuracy, increasing total cost.
• Factors like client selection, data heterogeneity (non-IID), and optimizer choice heavily influence the rate.
• Communication-efficient techniques often trade-off a slightly slower per-round convergence for a much lower cost per round, optimizing for total cost-to-accuracy.

A cryptographic protocol that allows a central server to compute the sum of client model updates without inspecting any individual client's contribution. While crucial for privacy, it adds communication overhead:

• Clients must exchange cryptographic masks or keys in addition to their model updates.
• The protocol must be robust to client dropouts during the aggregation phase.
• Advanced schemes aim to provide information-theoretic privacy with minimal additional communication burden compared to plaintext aggregation.

The canonical federated learning algorithm where communication cost is a first-order concern. Its parameters directly control cost:

• Local Epochs (E): Number of local training passes before communicating. Higher E reduces communication frequency but can cause client drift.
• Client Fraction (C): Proportion of clients selected per round. Lower C reduces per-round bandwidth but may slow convergence.
• Batch Size (B): Affects local computation speed and the stochastic nature of updates. Tuning these hyperparameters is a primary method for managing the total communication budget.