Differential Privacy

The output of a differentially private analysis is statistically indistinguishable whether or not any single individual's data was included. This is achieved by injecting calibrated noise scaled to the sensitivity of the query.

• An attacker observing the output cannot confidently determine if a specific record was used.
• The guarantee holds even against adversaries with unlimited auxiliary information.
• This is the foundational property that defeats membership inference attacks.

Privacy loss is bounded by a parameter epsilon (ε), known as the privacy budget. A smaller epsilon (e.g., 0.1) provides stronger privacy by adding more noise, while a larger epsilon (e.g., 10) provides weaker privacy with higher utility.

• ε quantifies the maximum divergence between outputs on adjacent datasets.
• The privacy budget is composable; it is consumed across sequential queries.
• A typical target range for strong privacy is ε ≤ 1.

Any arbitrary computation applied to the output of a differentially private mechanism cannot weaken the privacy guarantee. The output remains differentially private without requiring any additional noise.

• An attacker cannot 'unlock' privacy by transforming the data.
• This allows safe release of statistics for downstream analysis.
• Holds true for any deterministic or randomized function applied after the noise injection.

When multiple differentially private computations are performed on the same dataset, the total privacy loss is the sum of their individual epsilon values. This enforces a strict, predictable degradation of privacy over repeated queries.

• Running a query with ε=0.5 twice consumes a total budget of ε=1.0.
• System architects must track this privacy budget to prevent total leakage.
• This property necessitates query auditing and budget management systems.

When differentially private queries are applied to disjoint subsets of a dataset, the total privacy cost is the maximum epsilon among the queries, not the sum. This enables efficient use of the privacy budget in partitioned data architectures.

• Partitioning data by user ID allows unlimited queries per user without budget stacking.
• Critical for scaling privacy-preserving analytics across isolated data silos.
• Exploits the fact that an individual's data resides in only one partition.

Differential privacy naturally extends protection to groups. If a mechanism provides ε-differential privacy for a single individual, it provides kε-differential privacy for a group of size k.

• Protects against adversaries trying to infer properties of families or cohorts.
• The privacy guarantee degrades linearly with the size of the group being targeted.
• This is a direct consequence of the mathematical definition, not an add-on feature.

The epsilon (ε) parameter quantifies the total allowable privacy loss. A smaller epsilon (e.g., 0.1) provides a stronger guarantee but adds more noise, while a larger epsilon (e.g., 10) offers weaker privacy but higher utility. The budget is consumed cumulatively across all queries, and once exhausted, no further analysis is permitted. This is formalized as the sequential composition theorem, where the total privacy loss is the sum of individual epsilons.

The maximum change in a function's output when a single record is added or removed from the dataset. Global sensitivity measures the worst-case difference across all possible neighboring datasets. For a counting query, sensitivity is 1; for a sum query bounded by C, sensitivity is C. The noise magnitude scales proportionally to sensitivity divided by epsilon, making sensitivity the critical bridge between data and noise calibration.

The primary threat that differential privacy defends against. An attacker exploits a model's confidence scores or internal representations to reconstruct training data. For example, a facial recognition model can be inverted to generate images of individuals in the training set. Differential privacy provably bounds the attacker's ability to distinguish whether any specific individual was present, rendering inversion attacks statistically infeasible below a certain epsilon threshold.

Two deployment models with different trust assumptions. Central DP (or global DP) trusts a curator to collect raw data and add noise to query outputs—this is the standard model for DP-SGD. Local DP adds noise on each user's device before data leaves, eliminating the need for a trusted curator but requiring significantly more noise to achieve the same epsilon. Google's RAPPOR and Apple's iOS telemetry use local DP.