Per-Sample Gradient Clipping

The core purpose is to enforce a hard upper limit on how much any single data point can affect the model update. Without clipping, an outlier could contribute a disproportionately large gradient, creating a privacy vulnerability. The operation computes the L2 norm of each per-sample gradient and, if it exceeds a predefined threshold C, scales it down to exactly C. This ensures that the sensitivity—the maximum change in the output from adding or removing one record—is strictly bounded by C.

The hyperparameter C defines the maximum allowed L2 norm for any individual gradient. Its selection is a critical trade-off:

• Too low: Excessive clipping destroys useful information, leading to slow convergence and poor model utility.
• Too high: The sensitivity bound becomes large, requiring more noise to achieve the same privacy guarantee, which also degrades utility. Tuning C is often done by monitoring the distribution of unclipped gradient norms during a non-private training run to find a reasonable percentile.

While a flat clipping threshold applies the same C to all layers, this can be suboptimal because gradient norms often vary significantly across different parts of the network. Adaptive clipping strategies dynamically adjust the threshold per layer or per parameter group based on their historical norm statistics. This preserves more information from layers with naturally small gradients while still bounding the overall sensitivity, often yielding better model accuracy for the same privacy budget.

Standard SGD clips the gradient of the entire batch, which is a single vector. Per-sample clipping requires computing the gradient for each example independently before averaging. Naively, this would be extremely slow. Modern frameworks like Opacus implement efficient per-sample gradient computation using hooks that compute per-sample norms from the per-sample activation and output gradients without materializing every individual gradient tensor, making the overhead manageable for deep networks.

Clipping is the first half of the DP-SGD privacy guarantee. After per-sample gradients are clipped to bound sensitivity to C, the aggregated batch gradient has calibrated Gaussian noise added to it. The standard deviation of this noise is proportional to C / batch_size. A smaller clipping threshold directly enables adding less noise for the same privacy level, illustrating the direct coupling between the clipping operation and the final noise multiplier.

A memory-efficient implementation that avoids instantiating per-sample gradients in memory. Instead of computing the gradient for each sample and then clipping, ghost clipping computes the per-sample gradient norm directly from the per-sample activations and the output gradient of the loss. The scaling factor is then applied implicitly during the backward pass. This reduces peak memory usage from O(batch_size * num_params) to O(num_params), making DP-SGD feasible for large models.