Privacy Accounting

The privacy budget is the total allowable privacy loss, quantified by the parameters epsilon (ε) and delta (δ).

• Epsilon (ε): Bounds the multiplicative difference in the probability of any output. A lower ε provides stronger privacy.
• Delta (δ): A small probability of a complete privacy failure, representing an additive slack. It is typically set to a value less than the inverse of the dataset size. The budget is allocated at the project's outset, and the privacy account tracks expenditure against this limit.

For each mechanism (e.g., a training round with noise addition), a Privacy Loss Random Variable quantifies the actual privacy cost incurred. Its distribution depends on the mechanism's noise scale and the specific query. The moments accountant and Rényi Differential Privacy (RDP) track these variables to enable tight composition, providing a more accurate measure of cumulative loss than simple sequential composition of (ε, δ) bounds.

Composition theorems mathematically define how privacy loss accumulates over multiple queries or training rounds.

• Basic Sequential Composition: The epsilons sum, and the deltas sum. This is simple but yields loose, pessimistic bounds.
• Advanced Composition: Provides tighter bounds, especially for many iterations.
• Rényi Composition: Under RDP, Rényi divergences of the same order α add, allowing for exceptionally precise accounting, which is then converted to a final (ε, δ)-DP guarantee.

Each privacy-preserving operation consumes budget at a rate determined by its internal parameters.

• Noise Scale (σ): For the Gaussian mechanism, the standard deviation of the added noise relative to the query's L2 sensitivity. A larger σ consumes less budget per round.
• Sampling Rate (q): In federated learning, the fraction of clients selected per round. Privacy amplification theorems (like amplification by subsampling) reduce the effective privacy cost when q < 1.
• Clipping Norm (C): The maximum L2 norm bound applied to individual client updates before noise addition, which defines the sensitivity of the aggregation query.

The spending policy is an algorithm that decides how much budget to allocate per training round. It can be:

• Fixed: Allocates a uniform ε per round.
• Adaptive: Allocates more budget in later rounds for fine-tuning, or varies it based on convergence metrics. The stopping criterion is triggered when the remaining privacy budget falls below a threshold, halting training to prevent budget overrun. This enforces the formal guarantee.

A cryptographically verifiable audit log is a core component for compliance and trust. It immutably records:

• Initial budget allocation (ε_total, δ_total).
• Per-round parameters: noise scale, clipping norm, client sampling rate.
• Calculated privacy expenditure per round.
• Cumulative expenditure and remaining budget. This log allows external auditors or regulators to verify that the system adhered to its declared privacy policy throughout training.

Differential Privacy (DP) is the foundational mathematical framework for privacy accounting. It provides a rigorous, quantifiable guarantee that the inclusion or exclusion of any single individual's data in a dataset has a negligible effect on the output of a computation (e.g., a model update).

• Core Mechanism: Achieved by injecting calibrated noise (e.g., via the Gaussian or Laplace mechanism) into computations.
• Privacy Parameters: Defined by (ε, δ). Epsilon (ε) bounds the multiplicative privacy loss; a lower ε means stronger privacy. Delta (δ) is a probability of the privacy guarantee failing.
• Role in Accounting: Serves as the 'unit of measurement' for privacy expenditure. Each noisy operation (like a training round) consumes a portion of the total privacy budget.

Rényi Differential Privacy (RDP) is a relaxation of pure (ε,0)-DP that uses Rényi divergence for analysis. It is the dominant tool for advanced privacy accounting due to its superior composition properties.

• Tighter Composition: When multiple DP operations are composed (chained together), RDP provides much tighter bounds on the total privacy loss compared to the basic Advanced Composition Theorem.
• Conversion to (ε, δ): RDP bounds are typically finalized by converting them back to the standard (ε, δ)-DP guarantee for final reporting.
• Practical Use: Most modern DP libraries (like Google's DP-SGD) and federated learning frameworks use RDP for tracking privacy loss across thousands of training iterations.

The Privacy Budget is a pre-defined, maximum allowable amount of cumulative privacy loss (ε_total, δ_total) for an entire machine learning training process. It is the core resource managed by privacy accounting.

• Allocation Strategy: The total budget must be strategically allocated across training rounds, hyperparameter tuning, and final model evaluation.
• Exhaustion: Once the budget is consumed, the training process must stop to avoid violating the privacy guarantee. This creates a fundamental privacy-utility trade-off.
• Example: A system might be configured with a total budget of (ε=8.0, δ=1e-5). Privacy accounting tracks the expenditure after each federated round, ensuring the sum never exceeds these limits.

The Gaussian Mechanism is the primary method for achieving (ε, δ)-differential privacy by adding noise drawn from a Gaussian (normal) distribution to a function's output. It is central to private federated learning.

• Noise Calibration: The scale of the noise (σ) is proportional to the function's L2 sensitivity (the maximum change in output given a change in one input record) and inversely proportional to the desired ε and δ.
• Use in FL: Applied to clipped model updates (gradients) before they are sent from a client or during secure aggregation on the server. Each noisy update has a known privacy cost.
• Accounting Link: The privacy cost of each application of the Gaussian mechanism is calculated and added to the running total by the privacy accountant.

The Moments Accountant is a specific, highly efficient privacy accounting algorithm that leverages the Rényi DP framework. It was popularized by the DP-SGD algorithm and is now standard for iterative training.

• Mechanism: It tracks a moment generating function (or Rényi moments) of the privacy loss random variable across all training iterations.
• Efficiency: Provides significantly tighter privacy loss bounds compared to naive composition, allowing for more training rounds within the same fixed privacy budget.
• Implementation: It is an 'accountant' in the software sense—a module that takes the noise multiplier (σ) and sampling rate for each round as input and outputs the cumulative (ε, δ) after any number of steps.

Local Differential Privacy (LDP) is a distributed model where each client perturbs their own data before sending it to the aggregator. This contrasts with the central model of DP used in most federated learning privacy accounting.

• Stronger Client Guarantee: Provides privacy even against a malicious or curious central server, as the server only ever sees noisy data.
• Different Accounting: Privacy is enforced at the source. The server's aggregation of already-private reports generally does not incur additional privacy loss, simplifying the accounting perspective.
• Trade-off: LDP typically requires significantly more noise per client to achieve the same privacy level, often resulting in lower model utility compared to the central DP model with secure aggregation.