The moments accountant is the core algorithmic innovation that made differentially private stochastic gradient descent (DP-SGD) practical for deep learning. Instead of using the simple but loose composition theorems of standard differential privacy, it computes the log of the moment-generating function of the privacy loss random variable at each training step. By evaluating this function across a range of moments and optimizing over them, the accountant converts the sequence of per-step Gaussian noise additions into a single, tight (ε, δ) guarantee. This method, introduced by Abadi et al. in their seminal 2016 paper, provides significantly smaller epsilon values than advanced composition theorems, enabling meaningful privacy budgets for large-scale neural network training.
Glossary
Moments Accountant
What is a Moments Accountant?
A moments accountant is a privacy accounting technique that tracks the moment-generating function of the privacy loss random variable to compute tight, numerically accurate bounds on the total privacy cost of composed differentially private mechanisms.
The technique works by maintaining a running tally of higher-order moments of the privacy loss, specifically leveraging the subsampled Gaussian mechanism used in DP-SGD. For each lot sampled from the training set, the accountant computes the privacy cost analytically using the log moment-generating function, which has a closed-form expression for the Gaussian distribution. The final privacy parameters are derived by applying the tail bound theorem to convert the accumulated moments into a standard (ε, δ)-differential privacy guarantee. This approach is the default privacy accountant in frameworks like TensorFlow Privacy and Opacus, and its tightness is critical for accurately tracking the privacy budget without overestimating the true privacy loss.
Key Properties of the Moments Accountant
The Moments Accountant is a precise privacy accounting technique that tracks the moment-generating function of the privacy loss random variable to compute tight, non-trivial bounds on the overall privacy cost of composed mechanisms like DP-SGD.
Tighter Composition Bounds
The Moments Accountant provides significantly tighter bounds on cumulative privacy loss compared to the standard strong composition theorem. By tracking the log of the moment-generating function of the privacy loss random variable at multiple orders, it avoids the linear overestimation of epsilon that plagues simpler composition methods. This allows for more training iterations under the same privacy budget.
Moment-Generating Function Tracking
Instead of tracking a single epsilon value, the accountant maintains a function α(λ) — the log of the λ-th moment of the privacy loss random variable. For each step of DP-SGD, it computes the worst-case contribution to this function. The final privacy guarantee is derived by optimizing over all moments λ to find the tightest possible (ε, δ) pair.
Subsampling Amplification Integration
The accountant natively incorporates the privacy amplification by subsampling effect. When training with DP-SGD, each batch is a random sample from the full dataset. The Moments Accountant precisely computes the privacy loss distribution of the subsampled Gaussian mechanism, capturing the non-trivial privacy boost that arises because a single record is only included in a batch with probability q.
Numerical Tail Bound Conversion
After accumulating the moments over all training steps, the accountant converts the moment bound α(λ) into a standard (ε, δ)-differential privacy guarantee. This is done by applying Markov's inequality to the moment-generating function: for any ε > 0, the probability that the privacy loss exceeds ε is bounded by exp(α(λ) - λε). The optimal ε is found by minimizing over λ.
Foundation of Modern DP-SGD
Introduced by Abadi et al. in their 2016 paper 'Deep Learning with Differential Privacy', the Moments Accountant was the breakthrough that made differentially private deep learning practical. Prior accounting methods forced epsilon to grow too quickly with the number of training steps, making deep models infeasible. The Moments Accountant enabled training deep neural networks with meaningful privacy guarantees.
Relationship to Rényi DP
The Moments Accountant is closely related to Rényi Differential Privacy (RDP). The α(λ) function tracked by the accountant is essentially the RDP privacy parameter ε_α at order α = λ + 1. RDP provides a cleaner algebraic framework for composition, and many modern implementations use RDP accounting as a direct successor to the Moments Accountant, converting to (ε, δ)-DP only at the final reporting step.
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
Frequently Asked Questions
Explore the technical nuances of the Moments Accountant, the gold-standard privacy accounting technique that provides tight, composable bounds for differentially private machine learning.
A Moments Accountant is a specific privacy accounting technique used in Differentially Private Stochastic Gradient Descent (DP-SGD) to compute a tight upper bound on the total privacy loss (ε) of a composed mechanism. Unlike naive composition theorems that linearly sum the ε values of each step, the Moments Accountant tracks the logarithm of the moment-generating function of the privacy loss random variable. By evaluating this function at multiple orders (λ), it accurately captures the tail-bound behavior of the cumulative privacy loss distribution. This method, introduced by Abadi et al. in 2016, leverages the fact that the privacy loss of subsampled Gaussian mechanisms is a subgaussian random variable, allowing for significantly tighter composition bounds than the standard advanced composition theorem.
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us