Differential Privacy (DP)

The Laplace Mechanism is the foundational algorithm for achieving ε-differential privacy for numeric queries (e.g., counts, sums, averages). It works by adding noise drawn from a Laplace distribution to the true query result. The scale of the noise is calibrated to the query's global sensitivity—the maximum possible change in the output when any single individual's data is added or removed—divided by the privacy budget ε.

• Key Property: Provides pure (ε,0)-DP.
• Example: Publishing the average salary in a company database. The mechanism calculates the true average, then adds Laplace noise. The amount of noise ensures that observing the published result does not reveal with high confidence whether any specific employee's record was included.

The Gaussian Mechanism is used to achieve the relaxed (ε, δ)-differential privacy guarantee. It adds noise drawn from a Gaussian (normal) distribution to query outputs. This mechanism is often preferred for its compatibility with many analytical tools and because the sum of Gaussian noises remains Gaussian.

• Key Property: Provides (ε, δ)-DP, where δ represents a small probability of privacy guarantee failure.
• When to Use: For complex, iterative algorithms like stochastic gradient descent in machine learning, where the cumulative privacy loss across many steps is analyzed using composition theorems. The Gaussian mechanism typically allows for tighter privacy accounting in these scenarios compared to the Laplace mechanism.

The Exponential Mechanism is designed for non-numeric queries where the output is a discrete object, such as selecting the best candidate from a set (e.g., 'which disease is most prevalent?' or 'what is the optimal price?'). Instead of adding noise directly, it randomly selects an output from the set of all possible outputs, with a probability exponentially weighted by a quality (or utility) score.

• How it works: Outputs with higher utility scores (better answers) are exponentially more likely to be chosen. The sensitivity of the utility function determines the level of randomness.
• Example: Choosing the most common medical diagnosis in a private dataset. The mechanism will randomly pick a diagnosis, but the true most common one has the highest probability of being selected, ensuring both utility and privacy.

The privacy budget, ε (epsilon), is the central parameter quantifying the maximum allowable privacy loss. A smaller ε provides a stronger privacy guarantee but requires more noise, reducing data utility. The privacy loss random variable measures the actual leakage for a specific outcome.

• Interpretation: ε can be loosely interpreted as a bound on the log-likelihood ratio of seeing an output with or without any individual's data. An ε of 0.1 implies outputs are very similar in both cases; an ε of 10 implies they could be significantly different.
• Budget Management: In complex analyses, the total ε is divided (composed) across multiple queries or training steps. Advanced composition theorems (Basic, Advanced, or using the Moments Accountant) track the cumulative privacy expenditure to ensure the total does not exceed the allocated budget.

These are two principal models defining where noise is added in the data pipeline.

• Local Differential Privacy (LDP): Each user perturbs their own data on their device before sending it to the data collector. The collector never sees raw data.

  • Use Case: Apple's use in iOS for telemetry collection (e.g., emoji frequency).
  • Pro/Con: Provides a stronger trust model (untrusted server) but requires significantly more noise per user, hurting utility for small populations.

• Central Differential Privacy (CDP): A trusted curator (the server) holds the raw dataset and applies DP mechanisms to query results before release.

  • Use Case: The U.S. Census Bureau's 2020 Census data release.
  • Pro/Con: Achieves much higher utility for the same ε but requires trusting the central entity to securely manage the raw data.

DP-SGD is the canonical algorithm for training machine learning models with differential privacy guarantees. It modifies standard SGD by:

1. Clipping Gradients: The L2 norm of each individual's gradient contribution is bounded by a clipping constant C. This limits the sensitivity of the update step.
2. Adding Noise: Gaussian noise is added to the average of the clipped gradients within each training batch.
3. Privacy Accounting: The privacy cost of each noisy update is tracked using composition (e.g., the Moments Accountant), yielding a final (ε, δ) guarantee for the fully trained model.

• Trade-off: The clipping and noise introduce a utility-privacy trade-off. Higher privacy (lower ε) typically results in lower model accuracy.
• Impact: Enables the training of models on sensitive data (e.g., medical records) with a mathematically proven guarantee against membership inference attacks.

In healthcare, DP enables collaborative research across hospitals without sharing sensitive Protected Health Information (PHI).

• Multi-institutional Studies: Researchers can train a model for disease prediction (e.g., cancer detection) by having each hospital add DP noise to its model's summary statistics before a secure merge.
• Clinical Trial Analysis: Statisticians can compute aggregate outcomes (e.g., average treatment effect) from participant data with a mathematical guarantee that an individual's participation status (in/out of the trial) is protected. This balances the utility of large-scale medical research with strict compliance with regulations like HIPAA and GDPR.

Banks and payment processors need to detect fraudulent transactions by analyzing patterns across millions of users. Using DP, they can:

• Build aggregate models of normal vs. anomalous transaction behavior without exposing any customer's specific financial history.
• Securely share indicators of compromise (IoCs) or fraud patterns between competing institutions by exchanging only differentially private statistics. This allows for collaborative defense against sophisticated financial crime syndicates while maintaining strict client confidentiality and adhering to financial regulations like GLBA.

Companies run thousands of A/B tests to evaluate feature changes. DP allows them to compute accurate experiment results (e.g., click-through rate lift) while preventing analysts from inferring individual user actions. By adding noise to the aggregate metrics (counts, sums, averages) used in experiment reports, DP ensures that seeing the report does not reveal whether a specific user was in the test group or what their action was. This is critical for internal governance, allowing product teams to make data-driven decisions without exposing raw user-level telemetry to a broad internal audience.

A form of encryption that allows computations to be performed directly on encrypted data. The results, when decrypted, match the output of operations performed on the plaintext. This enables privacy-preserving outsourcing of data processing to untrusted servers.

• Key Mechanism: Mathematical schemes (e.g., CKKS, BGV) that preserve algebraic structure.
• Primary Use Case: Secure cloud-based model inference on encrypted client data.
• Contrast with DP: HE provides cryptographic security (stronger guarantee) but is computationally intensive, whereas DP provides a statistical privacy guarantee and is more scalable for large-scale analytics.

A cryptographic protocol that enables multiple parties to jointly compute a function over their private inputs while keeping those inputs concealed from each other. No party learns anything beyond the final output of the computation.

• Key Mechanism: Secret sharing and garbled circuits to distribute computation.
• Primary Use Case: Privacy-preserving data collaboration between competing entities (e.g., banks detecting fraud without sharing customer records).
• Contrast with DP: MPC provides input privacy in a multi-party setting with a precise cryptographic guarantee. DP is often used for releasing aggregate statistics from a single trusted curator to the public.

A decentralized machine learning paradigm where the model is trained across multiple edge devices or servers holding local data samples, without exchanging the raw data itself. Instead, only model updates (e.g., gradients) are shared with a central server for aggregation.

• Key Mechanism: Iterative averaging of locally computed model updates (e.g., FedAvg).
• Primary Use Case: Training on sensitive, distributed data sources like mobile phones or hospital records.
• Synergy with DP: Federated Learning is often combined with Differential Privacy by adding noise to the shared model updates before aggregation, providing a defense against inference attacks on the updates.

The process of using rigorous mathematical methods to prove or disprove the correctness of a system's design with respect to a formal specification. In safety contexts, it is used to prove that a system will never enter an unsafe state.

• Key Mechanism: Model checking, theorem proving, and abstract interpretation.
• Primary Use Case: Verifying safety properties of controllers, neural networks (via bounded methods), or cryptographic protocols.
• Contrast with DP: Formal Verification provides deterministic guarantees about system behavior. DP provides a probabilistic guarantee about privacy loss. Both are mathematical frameworks for rigorous assurance.

An extension of the standard Markov Decision Process (MDP) framework that incorporates constraints on expected cumulative costs. It is the foundational model for Safe Reinforcement Learning (Safe RL), where an agent must learn a policy that maximizes reward while satisfying safety constraints.

• Key Mechanism: Augments the MDP tuple with a cost function and a cost constraint threshold.
• Primary Use Case: Formulating robotics or autonomous system tasks where certain failures (e.g., collisions) must be bounded.
• Relation to Safety: While DP protects data privacy, CMDPs formalize physical safety constraints for sequential decision-making agents operating in environments.

The task of identifying whether a new input data point is statistically different from the training data distribution. This is critical for safety in deployed ML systems to prevent high-confidence predictions on unfamiliar, potentially risky inputs.

• Key Mechanism: Using metrics like prediction uncertainty, feature-space distance, or likelihood from a generative model.
• Primary Use Case: Triggering human intervention or a safe fallback policy when a self-driving car encounters a novel object.
• Synergy with Safety: Both OOD Detection and Differential Privacy are critical components of a robust AI safety posture—one addresses operational unknowns, the other addresses data confidentiality.