Output Perturbation

Output perturbation applies noise after the model has been fully trained, requiring no modification to the training pipeline or architecture. This makes it ideal for retrofitting privacy onto existing deployed models without retraining costs.

• Operates on the prediction vector (softmax output) or aggregate query results
• No access to model internals or training data required at perturbation time
• Compatible with any black-box model, including proprietary third-party APIs

The core mechanism adds random noise drawn from a statistical distribution—typically Laplace or Gaussian—calibrated to the sensitivity of the query function. The noise scale is governed by the privacy parameter epsilon.

• Laplace mechanism: Optimal for pure epsilon-differential privacy with L1 sensitivity
• Gaussian mechanism: Preferred for (epsilon, delta)-differential privacy with L2 sensitivity
• Noise scale = sensitivity / epsilon, creating a direct privacy-utility trade-off

Global sensitivity measures the maximum change in the query output when a single record is added or removed from the dataset. This metric determines the minimum noise required to achieve a given privacy guarantee.

• For count queries, sensitivity = 1 (a single record changes the count by at most 1)
• For prediction vectors, sensitivity depends on the L1 or L2 norm of the output space
• Lower sensitivity enables stronger privacy with less utility degradation

Each perturbed query consumes a portion of the total privacy budget (epsilon). A privacy accountant tracks cumulative epsilon expenditure across all queries to ensure the overall guarantee remains within acceptable bounds.

• Sequential composition: querying k times consumes k * epsilon budget
• Parallel composition: queries on disjoint data subsets consume only the maximum single-query epsilon
• Budget exhaustion triggers query denial or increased noise levels

By perturbing confidence scores before returning them to the user, output perturbation directly disrupts model inversion attacks that rely on precise probability gradients to reconstruct training data.

• Noisy confidence scores degrade gradient-based reconstruction fidelity
• Top-k masking combined with perturbation further limits information leakage
• Effective against both white-box and black-box inversion adversaries

The fundamental tension in output perturbation is between answer accuracy and privacy protection. Higher noise (lower epsilon) provides stronger guarantees but may render predictions useless for downstream decisions.

• Epsilon < 1: Strong privacy, significant utility loss for complex queries
• Epsilon 1-10: Moderate privacy, acceptable for aggregate statistics
• Epsilon > 10: Weak privacy guarantees, minimal utility impact
• Optimal calibration depends on the sensitivity of the downstream application

The primary threat that output perturbation defends against. An adversary exploits access to a model's prediction vector to reconstruct representative samples of a target class. By iteratively optimizing a random input to maximize the confidence score for a specific label, attackers can recover recognizable faces or sensitive attributes. Output perturbation disrupts this optimization by injecting calibrated noise into the confidence scores, making the gradient signal unreliable for reconstruction.

The mathematical framework that provides formal guarantees for output perturbation. DP ensures that the probability distribution of a model's output is nearly identical whether or not any single individual's data was included in the training set. The privacy budget (epsilon) quantifies the maximum information leakage. Output perturbation is a specific instantiation of DP applied at the inference stage rather than during training.

A related privacy attack that determines whether a specific record was in the training set by analyzing prediction confidence. Models typically exhibit higher confidence on training data. Output perturbation with sufficient noise masks these overfitting signals by flattening the confidence distribution, making it statistically impossible to distinguish members from non-members at a meaningful rate.

A lightweight alternative to noise injection that truncates the prediction vector to return only the top-k class labels without probability scores. While simpler to implement than calibrated perturbation, masking provides no formal privacy guarantee. It is often combined with output perturbation as a defense-in-depth measure: first mask to limit information, then perturb the remaining scores to achieve a provable epsilon guarantee.

Differentially Private Stochastic Gradient Descent applies privacy protection during training rather than inference. It uses per-sample gradient clipping and Gaussian noise injection to bound the influence of individual training examples. While output perturbation protects only the final prediction, DP-SGD protects the model parameters themselves. The two techniques are complementary and can be combined for end-to-end privacy guarantees.

The raw material that output perturbation transforms. A prediction vector is the softmax output of a classifier containing confidence scores for each class. These scores encode subtle memorization patterns that inversion and membership attacks exploit. Output perturbation adds noise calibrated to the sensitivity of this vector—the maximum change in output caused by modifying a single training record—to mask individual contributions.