Differential Privacy (DP)

The epsilon (ε) parameter is the core privacy budget that quantifies the maximum allowable privacy loss. It defines the strength of the guarantee: a smaller ε provides stronger privacy but typically reduces the utility (accuracy) of the model's output. The formal definition ensures that for any two adjacent datasets (differing by one record), the probability of any output is within a multiplicative factor of e^ε.

• Example: A common ε value for research is 1.0 or 8.0. Deployments in highly sensitive domains like healthcare may use ε < 1.0.
• Trade-off: This creates the fundamental utility-privacy trade-off. Tuning ε is critical for balancing acceptable accuracy with rigorous privacy protection.

The Laplace Mechanism is a fundamental algorithm for achieving differential privacy for numerical queries (e.g., calculating an average or sum). It works by adding carefully calibrated noise drawn from a Laplace distribution to the true query result. The scale of the noise is proportional to the sensitivity of the query (the maximum change the query can have when one record changes) divided by ε.

• Key Formula: Noisy Output = True Query Result + Laplace(Δf / ε)
• Sensitivity (Δf): For a counting query, Δf = 1. For an average, sensitivity depends on the data bounds.
• Use Case: Adding private noise to aggregated model update statistics (e.g., gradients or weights) in federated learning.

The Gaussian Mechanism is an alternative to the Laplace mechanism that adds noise drawn from a Gaussian (normal) distribution. It is often preferred in machine learning contexts, particularly for high-dimensional data like gradients, because the noise characteristics can be more favorable for optimization. It requires defining an additional parameter, delta (δ), which represents a small probability of privacy guarantee failure.

• Key Difference: Provides (ε, δ)-differential privacy, a slightly relaxed guarantee compared to pure ε-DP.
• Use Case: The standard mechanism for implementing Differentially Private Stochastic Gradient Descent (DP-SGD), where Gaussian noise is added to clipped gradients during model training.

This distinction defines where the privacy mechanism is applied in a data pipeline.

• Local Differential Privacy (LDP): Each data owner perturbs their own data before sending it to the aggregator. This provides a strong trust model—the aggregator never sees raw data—but requires more noise per individual, reducing aggregate utility. Used in scenarios like Google's RAPPOR for browser data collection.
• Central Differential Privacy (CDP): A trusted curator holds the raw dataset and applies the privacy mechanism before releasing query results or a model. This allows for less noise overall (better utility) but requires trusting the central entity. Federated learning with secure aggregation can be viewed as a pathway to emulate a trusted curator.

Composition rules are essential for analyzing the total privacy cost of a complex algorithm that makes multiple queries to the data. They dictate how the privacy budgets (ε) accumulate.

• Sequential Composition: If you perform k DP mechanisms, each with privacy parameters (ε_i), the total privacy cost is the sum of the ε_i. This is a conservative bound.
• Advanced Composition: Provides a tighter (more favorable) bound for the composition of many mechanisms, especially the Gaussian mechanism, often scaling with the square root of the number of steps.
• Importance: These theorems enable the privacy accounting for iterative algorithms like DP-SGD, where gradients are queried in thousands of training steps. Tools like Google's TensorFlow Privacy and Opacus perform automatic composition accounting.

DP-SGD is the canonical algorithm for training deep learning models with differential privacy. It modifies standard SGD with three key steps:

1. Per-Sample Gradient Clipping: The gradient for each individual training example is clipped to a maximum L2 norm. This bounds the sensitivity (Δf).
2. Noise Addition: Gaussian noise is added to the average of the clipped gradients for the batch.
3. Privacy Accounting: The privacy cost (ε, δ) is meticulously tracked across all training steps using composition theorems.

• Impact: This ensures the final model parameters do not reveal specifics about any individual training example.
• Federated Learning Link: In federated learning, DP-SGD can be applied locally on client devices before updates are sent, or centrally to the aggregated update on the server.

Epsilon (ε) is the core privacy loss parameter in differential privacy, quantifying the maximum allowable difference in the probability of any output when a single individual's data is included or excluded from the dataset. A smaller ε provides stronger privacy guarantees but typically reduces model utility (accuracy).

• Interpretation: ε = 0.1 offers very strong privacy; ε = 10 offers weaker, more permissive privacy.
• Composition: The total privacy budget ε_total accumulates over multiple queries or training rounds.
• Trade-off: Tuning ε is the primary mechanism for managing the utility-privacy trade-off.

Secure Aggregation is a cryptographic protocol that allows a central server in federated learning to compute the sum (or average) of client model updates without being able to inspect any individual client's contribution. It provides input privacy, preventing the server from learning a specific client's gradient.

• Mechanism: Often uses multi-party computation (MPC) or homomorphic encryption.
• Complement to DP: While secure aggregation hides individual updates, DP adds noise to the aggregated result to provide formal, mathematical privacy guarantees against inference attacks, even if the aggregated sum is revealed.

The utility-privacy trade-off describes the fundamental inverse relationship in privacy-preserving ML: increasing the strength of privacy guarantees (e.g., by lowering ε or adding more noise) typically reduces the final model's accuracy, convergence speed, or overall utility.

• Core Challenge: The goal is to find an operating point where privacy is formally guaranteed while model performance remains acceptable for the task.
• Quantification: Measured by plotting model accuracy (or loss) against the ε budget.
• Management: Techniques like adaptive clipping and privacy accounting (e.g., the Moments Accountant) help optimize this trade-off.

Local Differential Privacy (LDP) is a variant where each data point is randomized with DP guarantees before being sent to a central server. This provides a stronger trust model, as the curator never sees raw data.

• Contrast with Central DP: In standard (central) DP, trusted curator applies noise after collecting raw data.
• Use in Federated Learning: LDP can be applied directly to client updates, but often requires significantly more noise than central DP applied after secure aggregation, leading to a more severe utility loss.

The Gaussian and Laplace Mechanisms are the two primary noise-adding techniques for achieving differential privacy. They inject calibrated random noise into the output of a computation.

• Laplace Mechanism: Adds noise drawn from a Laplace distribution. Used for ε-DP guarantees on numeric queries. The scale of noise is Δf/ε, where Δf is the sensitivity.
• Gaussian Mechanism: Adds noise drawn from a Gaussian distribution. Used for (ε, δ)-DP guarantees. Often preferred in deep learning due to the properties of Gaussian noise during optimization.
• Sensitivity (Δf): The maximum change in a function's output when a single data point is altered. It directly determines the amount of noise required.

A Membership Inference Attack (MIA) is a privacy attack where an adversary, given access to a trained model and a target data record, attempts to determine if that record was part of the model's training set. DP is a primary defense.

• Vulnerability: Models can memorize specific training examples, making them vulnerable to MIAs.
• DP as Defense: By guaranteeing that the model's output is nearly indistinguishable regardless of any single record's presence, DP provably bounds an attacker's ability to succeed at MIA.
• Evaluation: The success rate of MIAs is a practical, empirical measure of a model's privacy leakage, complementing the theoretical ε guarantee.