Moments Accountant

Instead of naively summing ε values, the Moments Accountant computes the log of the moment-generating function of the privacy loss random variable at each training step. This captures the entire distribution of possible privacy losses, not just a single worst-case bound.

• Tracks moments across all orders λ to find the tightest bound
• Accounts for the subsampled Gaussian mechanism used in DP-SGD
• Converts accumulated log moments to a final (ε, δ) guarantee via tail bounds

The accountant leverages the connection between Rényi Differential Privacy (RDP) and the log moments of privacy loss. By computing the Rényi divergence of the subsampled Gaussian mechanism at each order α, it achieves composition bounds that are asymptotically tight.

• Avoids the looseness of basic composition theorems
• Provides significantly smaller ε values than the strong composition theorem
• Enables practical privacy budgets for deep models with hundreds of epochs

A critical innovation is the analytical integration of privacy amplification by subsampling into the moment computation. The accountant computes the exact log moment of the subsampled mechanism, accounting for the random batch selection in DP-SGD.

• Handles Poisson subsampling and uniform sampling without replacement
• Amplification factor depends on the sampling rate q = B/N
• Smaller batch sizes yield stronger amplification and lower privacy cost

The Moments Accountant is implemented as a numerical routine that runs alongside training. It maintains a running computation of the worst-case privacy loss at each step, enabling real-time budget monitoring.

• Computes the log moment for each integer order λ up to a maximum value
• Uses binary search to find the optimal δ for a target ε or vice versa
• Outputs the final (ε, δ) pair after all training steps complete

The Moments Accountant provides significantly tighter bounds than the advanced composition theorem of Dwork et al. For a typical DP-SGD run on MNIST with ε = 8, the strong composition theorem would report ε ≈ 150 — a catastrophic overestimate.

• Reduces reported ε by an order of magnitude or more
• Makes differential privacy feasible for deep learning in practice
• The tightness comes from directly analyzing the privacy loss distribution

The Moments Accountant has been largely superseded by the Fourier Accountant and Privacy Loss Distribution (PLD) methods, which provide even tighter bounds by working directly with the characteristic function of the privacy loss distribution.

• PLD accountant uses fast Fourier transform for numerical convolution
• Provides the tightest known composition bounds for DP-SGD
• Still conceptually grounded in the moment-tracking framework

A privacy definition based on the Rényi divergence that provides a natural framework for the Moments Accountant. The accountant tracks the logarithm of the moment-generating function of the privacy loss random variable evaluated at specific orders (α), which directly corresponds to computing the RDP guarantee. This relationship allows for tight composition of the Gaussian mechanism across thousands of DP-SGD iterations.

A privacy framework that characterizes a mechanism by its trade-off function between Type I and Type II errors in a hypothesis testing problem. GDP provides a central limit theorem for privacy composition, showing that the overall privacy loss of many composed mechanisms converges to a Gaussian trade-off function. This offers an alternative, operationally meaningful way to interpret the tight bounds computed by the Moments Accountant.

The probability distribution of the privacy loss random variable, which fully characterizes a mechanism's privacy properties. The Moments Accountant works by computing the moment-generating function of this distribution. Modern numerical accountants, such as Fourier accounting, directly compute the convolution of privacy loss distributions for even tighter composition bounds, representing the state-of-the-art evolution beyond the Moments Accountant.

A formal result quantifying how the total privacy loss accumulates when multiple differentially private mechanisms are applied to the same dataset. The Moments Accountant provides a tighter composition bound than the basic or advanced composition theorems by tracking higher-order moments of the privacy loss variable. This allows it to prove that DP-SGD satisfies (ε, δ)-DP with a significantly smaller ε than naive composition would suggest.

A technique where the random selection of a data subset before applying a differentially private mechanism provides a stronger privacy guarantee. The Moments Accountant explicitly models the privacy amplification effect of Poisson subsampling or uniform shuffling used in DP-SGD. By computing the moment-generating function of the subsampled Gaussian mechanism, it achieves the tight bounds that make deep learning with differential privacy practical.