Differentially Private Computer Vision

The foundational preprocessing step that bounds the influence of any single image on the model update. Each per-example gradient's L2 norm is clipped to a fixed threshold C, limiting sensitivity before noise injection.

• Prevents outlier images from dominating the gradient signal
• Clipping threshold is a critical hyperparameter balancing privacy and utility
• Computationally expensive for vision models due to per-example backpropagation

DP-SGD injects calibrated Gaussian noise into the averaged, clipped gradients before each weight update. The noise scale σ is proportional to the clipping norm and inversely proportional to the target privacy budget ε.

• Noise magnitude grows with model size, challenging for large vision architectures
• Requires tight privacy accounting via Moments Accountant or RDP composition
• Directly prevents memorization of individual training images

Achieving meaningful differential privacy (ε < 8) on complex vision benchmarks like ImageNet typically incurs a significant accuracy drop compared to non-private baselines. The tradeoff is more severe for high-resolution images and fine-grained classification.

• Transfer learning from public pretrained models dramatically improves DP accuracy
• Larger batch sizes and more epochs partially compensate for noise
• Object detection and segmentation remain open challenges under tight privacy budgets

Even when classification heads are trained privately, the backbone feature extractor can memorize and leak sensitive visual patterns. Full-stack DP training must apply noise to all trainable parameters, including convolutional kernels and attention layers.

• Frozen pretrained backbones reduce trainable parameters and noise requirements
• Vision Transformers exhibit different memorization patterns than CNNs
• Intermediate feature maps can be analyzed for privacy leakage via inversion attacks

Combining federated learning with differential privacy enables training on decentralized image datasets—such as medical scans across hospitals—without centralizing raw pixels. Each client clips and noises its local gradients before transmission.

• Protects against gradient inversion attacks that reconstruct training images
• User-level DP guarantees protect all images belonging to a single individual
• Communication efficiency becomes critical with added noise variance

Differentially Private GANs train the discriminator with noisy gradients, enabling the generator to produce realistic synthetic images that preserve dataset-level statistics without exposing real individuals.

• DP-GANs struggle with high-resolution outputs due to noise accumulation
• DP diffusion models are emerging as a more stable alternative
• Synthetic data can be shared freely for downstream tasks without privacy budget exhaustion

The privacy budget (ε, δ) is a finite, quantifiable limit on total information leakage. Each training epoch or query consumes a portion of this budget. Composition theorems govern how this budget degrades:

• Basic Composition: Privacy loss sums linearly across k mechanisms (ε_total = k * ε).
• Advanced Composition: Provides a tighter, sub-linear bound using the square root of k, essential for iterative training.
• Moments Accountant: Tracks higher-order moments of the privacy loss random variable, yielding the tightest bounds for DP-SGD in practice. In computer vision, managing the budget requires deciding how many epochs to train and at what noise multiplier, directly impacting the final model's utility.

Sensitivity defines the maximum change in a query's output from adding or removing a single record. In DP-SGD, gradient clipping enforces a bounded sensitivity by scaling down any per-example gradient whose L2 norm exceeds a threshold C. This is a critical hyperparameter:

• Too low: Gradients are overly distorted, destroying utility and preventing learning.
• Too high: Sensitivity is large, requiring more noise to achieve the same privacy level, which also degrades accuracy. For vision models, the distribution of gradient norms varies across layers and during training, making adaptive clipping techniques an active area of research.

A powerful phenomenon where randomly sampling a subset of data before applying a differentially private mechanism yields a stronger privacy guarantee than processing the full dataset. In DP-SGD, each batch is a random sample, and the privacy amplification factor is proportional to the sampling rate q = B/N (batch size / total dataset size).

• Smaller q provides greater amplification, allowing for a lower effective ε.
• This is why DP-SGD with small batches can achieve reasonable privacy-utility trade-offs.
• The Rényi Differential Privacy (RDP) framework provides a clean way to account for this amplification during composition.

Generative Adversarial Networks trained with differential privacy constraints on the discriminator's gradients. By adding noise to the discriminator updates during training, the generator learns to produce synthetic images without memorizing training examples. Key approaches include:

• DP-SGD on the discriminator: The generator receives privacy guarantees via post-processing immunity.
• DP-CGAN: Conditions generation on labels while preserving privacy.
• GS-WGAN: Uses the Wasserstein distance with a sanitized critic to improve stability. DP-GANs enable the creation of differentially private synthetic data for downstream vision tasks, though they often struggle with high-resolution, complex images.