Gradient Clipping

The primary mechanism of gradient clipping involves computing the L2 norm (Euclidean length) of the entire gradient vector. If this norm exceeds a predefined threshold C, the gradient is rescaled proportionally: g_clipped = g * min(1, C / ||g||₂). This preserves the direction of the update while capping its magnitude, preventing any single client from dominating the global model with an excessively large step. The threshold C is a critical hyperparameter—too low and learning stalls, too high and the defense becomes ineffective.

In adversarial federated learning scenarios, malicious clients can submit poisoned updates with arbitrarily large gradient magnitudes designed to corrupt the global model. Gradient clipping acts as a first line of defense by bounding the influence of any single update. When combined with Byzantine-resilient aggregation rules like Krum or Trimmed Mean, clipping ensures that even if an attacker submits an update with an infinite norm, its post-clipping contribution is limited to the threshold value, preventing catastrophic model corruption.

Gradient clipping is a fundamental building block of Differentially Private SGD (DP-SGD) and DP-FedAvg. To provide formal privacy guarantees, the sensitivity of each individual training example must be bounded. Clipping per-example gradients to a maximum L2 norm C explicitly defines this sensitivity bound. After clipping, calibrated Gaussian noise proportional to C is added. Without clipping, a single outlier data point could require infinite noise, destroying model utility. This makes clipping indispensable for privacy-preserving federated learning in healthcare.

Deep neural networks, particularly RNNs, LSTMs, and deep transformers, are susceptible to exploding gradients during training—a phenomenon where gradients grow exponentially through backpropagation. In federated settings with heterogeneous clinical data, non-IID distributions can exacerbate this instability. Gradient clipping provides a simple yet effective remedy by preventing gradient norms from exceeding a stable range, enabling the use of higher learning rates and accelerating convergence without the risk of numerical overflow or divergence.

Static clipping thresholds can be suboptimal as training progresses. Adaptive gradient clipping dynamically adjusts the threshold based on gradient statistics:

• Quantile-based clipping: Sets the threshold to a specific percentile (e.g., 90th) of observed gradient norms across clients
• Geometric median clipping: Uses the geometric median of client norms as an adaptive threshold
• Auto-clipping: Scales the threshold proportionally to the gradient norm history These methods reduce hyperparameter tuning burden and maintain optimal clipping tightness throughout training.

Two granularities of clipping exist in federated learning:

• Client-level clipping: Applied to the entire model update Δw before transmission to the server. Defends against malicious nodes but provides no per-example privacy guarantees.
• Per-example clipping: Applied to individual training example gradients during local SGD. Required for differential privacy but significantly increases computational overhead, as each example's gradient must be computed and clipped independently rather than in batched form. The choice depends on whether the primary goal is security, privacy, or computational efficiency.

The foundational aggregation algorithm that constructs a global model by computing a weighted average of locally trained model updates. Gradient clipping is a critical preprocessing step within FedAvg to bound the influence of any single client before averaging.

• Weights updates by local dataset size
• Prone to divergence with non-IID data
• Clipping stabilizes convergence in heterogeneous settings

A federated optimization framework that adds a proximal term to local objective functions to stabilize convergence across heterogeneous devices. While FedProx addresses objective inconsistency, gradient clipping complements it by handling system-level heterogeneity where resource-constrained clients might produce erratic updates.

• Penalizes deviation from the global model
• Tolerates variable local computation budgets
• Clipping prevents proximal term from being overwhelmed by outliers