Moments Accountant

Instead of tracking only the privacy loss (first moment), the Moments Accountant computes the moment generating function of the privacy loss random variable evaluated at multiple orders λ. This captures the full distributional tail behavior, enabling a tighter characterization of cumulative privacy loss across thousands of training iterations. The log of the moment generating function is bounded for each step, and these bounds are composed linearly across the entire training run.

The key advantage over strong composition theorems is the elimination of the √k factor in the δ term. For k iterations of the Gaussian mechanism with noise σ, the Moments Accountant proves that the composed mechanism satisfies (O(ε), δ)-DP with a significantly smaller ε than naive composition. This directly translates to higher utility for the same privacy guarantee, or stronger privacy for the same model accuracy.

The Moments Accountant is specifically designed for the subsampled Gaussian mechanism at the heart of DP-SGD. It analytically computes the privacy loss distribution for:

• Poisson subsampling of minibatches
• Gaussian noise addition to clipped gradients
• Sequential composition over training epochs This specialization yields bounds that are orders of magnitude tighter than generic composition theorems when applied to deep learning training loops.

The Moments Accountant is a precursor to Rényi Differential Privacy (RDP). Both track the moment generating function, but RDP formalizes the approach using Rényi divergence of order α. The Moments Accountant's λ parameter corresponds to RDP's α. RDP provides a cleaner functional perspective, while the Moments Accountant offers a direct algorithmic implementation for computing (ε, δ) guarantees from the moment bounds via tail bound conversion.

The accountant operates by maintaining a running log of moment bounds α_M(λ) for a discrete set of λ values. After each training step, it updates these bounds using the subsampled Gaussian mechanism's analytical formula. At the end of training, it converts the accumulated moment bounds into an (ε, δ) guarantee by solving:

• ε = min_λ (α_M(λ) - log δ) / λ This numerical optimization over λ yields the tightest provable privacy bound.

By providing tighter privacy accounting, the Moments Accountant directly enables training deeper models and longer training runs under a fixed privacy budget. Empirical results show that models trained with Moments Accountant bounds achieve significantly higher test accuracy compared to those constrained by advanced composition theorems, particularly in the low-ε regime (ε < 2) where privacy is strongest and every bit of budget efficiency matters.