Differential Privacy

The epsilon (ε) parameter quantifies the privacy loss. A smaller ε provides stronger privacy but less accuracy.

• ε = 0: Perfect privacy, zero information leakage.
• ε = 0.1 - 1.0: A strong, commonly targeted range for real-world deployments.
• ε > 10: Weak privacy guarantees, often insufficient for sensitive data. The budget is consumed with each query to the sensitive dataset. Once the total ε limit is reached, no further analysis is permitted to prevent complete reconstruction.

The composition theorem states that privacy loss accumulates linearly. If you query a dataset multiple times, the total epsilon is the sum of the individual epsilons.

• Mechanism: If analysis A is ε₁-DP and analysis B is ε₂-DP, releasing both results is (ε₁ + ε₂)-DP.
• Implication: This forces a hard limit on the number of queries allowed against a sensitive dataset, preventing the averaging out of noise through repeated sampling.

When queries operate on disjoint subsets of the data, the privacy cost does not accumulate. The total privacy loss is bounded by the maximum epsilon of any single query, not the sum.

• Example: If a dataset is partitioned by user ID, and a query runs independently on each partition, the overall privacy cost is equal to the cost of the query on a single partition.
• Benefit: This property is crucial for scalable systems, allowing many simultaneous computations without draining the privacy budget.

Any computation applied to the output of a differentially private mechanism does not degrade the privacy guarantee. Privacy is not a function of the output's utility.

• Formal Definition: If a mechanism M satisfies ε-DP, then for any arbitrary function f, f(M(x)) also satisfies ε-DP.
• Adversarial Resistance: An attacker cannot increase privacy loss by performing arbitrary transformations, machine learning, or statistical analysis on the noisy output. The raw data remains protected regardless of post-hoc computation.

The guarantee naturally extends to groups, but with a linear penalty. Protecting a group of size k weakens the guarantee by a factor of k.

• Mechanism: An ε-DP mechanism provides (k * ε)-DP for groups of size k.
• Correlation Risk: This property highlights the limitation of differential privacy against highly correlated data. If family members share genomic data, protecting one individual's privacy does not automatically provide a strong guarantee for the entire family unit.

The privacy guarantee holds even against adversarial analysts who choose their next query based on the results of previous queries. The privacy loss budget remains a valid upper bound regardless of the query strategy.

• Significance: This is a stronger property than non-adaptive composition. It proves that an attacker cannot strategically sequence queries to reverse-engineer the noise and extract more information than the total epsilon allows.

The privacy budget (ε) is the quantifiable limit on total privacy loss allowed over a series of queries on a sensitive dataset. A lower epsilon (e.g., ε=0.1) provides a stronger mathematical guarantee but introduces more noise, reducing utility. Privacy accountants track cumulative epsilon consumption to ensure the total leakage never exceeds a predefined threshold, after which further queries must be blocked to preserve the formal guarantee.

DP-SGD is the dominant algorithm for training deep learning models with differential privacy. It modifies standard SGD through two critical steps:

• Per-sample gradient clipping: Bounds the influence of any single training example by scaling down gradients whose L2 norm exceeds a threshold.
• Gaussian noise addition: Adds calibrated random noise to the aggregated gradients before updating model weights. This ensures the final model parameters do not memorize individual records.

A membership inference attack determines whether a specific data record was included in a model's training dataset by analyzing prediction confidence scores, loss values, or gradient behavior. Attackers exploit the tendency of models to be overconfident on memorized training examples. Differential privacy directly mitigates this by bounding the statistical influence of any single record, making membership indistinguishable from non-membership within the epsilon guarantee.

Secure aggregation is a cryptographic protocol that allows a central server to compute the sum of model updates from multiple clients without inspecting individual contributions. It uses secret sharing and masking techniques so the server only sees the aggregated result. When combined with differential privacy, secure aggregation provides a dual layer of defense: cryptography protects updates in transit, while DP noise masks residual patterns in the aggregated output.

Gradient leakage is an attack reconstructing private training data from publicly shared model gradients in distributed learning. Gradient inversion iteratively optimizes dummy inputs to produce gradients matching the observed ones, often recovering high-fidelity images or text. Differential privacy's noise injection directly degrades the signal-to-noise ratio required for successful inversion, making reconstruction mathematically infeasible below certain epsilon thresholds.

Homomorphic encryption (HE) enables computation directly on encrypted data without requiring decryption, producing an encrypted result that matches the computation on plaintext. While HE provides cryptographic confidentiality, it does not prevent inference-based leakage from outputs. Hybrid approaches apply DP noise before HE computation, ensuring that even if the encrypted result is later decrypted, individual records remain statistically protected.