Differentially Private Natural Language Processing

The critical preprocessing step that bounds the influence of any single training example. Before noise is added, the L2 norm of each individual gradient is computed and scaled down to a fixed threshold C if it exceeds it.

• Mechanism: grad = grad * min(1, C / ||grad||₂)
• Purpose: Limits the sensitivity of the training update, directly controlling how much a single document can affect the model.
• Impact on Text: Prevents the model from overfitting to unique phrases or rare tokens in a single training document.

The core perturbation step where random noise drawn from a Gaussian distribution is added to the aggregated, clipped gradients. The noise scale σ is calibrated to the clipping threshold and the desired privacy budget.

• Formula: Noise ~ N(0, (C * σ)²)
• Privacy Guarantee: Achieves (ε, δ)-differential privacy, where δ represents the small probability of a catastrophic privacy failure.
• Text-Specific Effect: Smooths out the model's ability to reproduce exact word sequences, acting as a forgetting mechanism for verbatim text.

A rigorous, composable ledger that tracks the total privacy loss ε consumed during iterative training. Each DP-SGD step incurs a cost, and training must halt once the allocated budget is exhausted.

• Moments Accountant: A specialized algorithm that provides tight, high-probability bounds on the cumulative privacy loss across thousands of training iterations.
• Rényi DP: An alternative accounting method using Rényi divergence that often yields tighter composition bounds than classical methods.
• Consequence: Forces a direct trade-off between the number of training epochs, model accuracy, and the formal privacy guarantee.

A privacy amplification technique where each training step operates on a randomly sampled mini-batch of the dataset rather than the full corpus. The randomness of the sampling process itself provides an additional layer of obfuscation.

• Amplification Effect: If a mechanism is ε-differentially private, applying it to a random q-fraction sample yields a stronger O(qε) privacy guarantee.
• Poisson Sampling: The gold standard where each data point is independently included in a batch with probability q, maximizing the amplification benefit.
• Benefit for NLP: Allows training on massive text corpora with a significantly lower effective privacy cost per example.

A resilience property guaranteeing that any arbitrary computation applied to the output of a differentially private mechanism cannot weaken the privacy guarantee. The model weights, once trained with DP-SGD, are safe to publish.

• Implication: The final model can be quantized, pruned, distilled, or deployed as a public API without any additional privacy risk to the training data.
• What it doesn't cover: This immunity does not protect against future attacks that query the model to infer training data (membership inference), but the underlying DP guarantee remains mathematically intact.
• Practical Use: Enables safe sharing of fine-tuned language models with partners or the open-source community.

A stronger privacy definition that protects all contributions from a single user, not just individual documents. This is critical for text datasets where a single author may have contributed multiple emails, reviews, or messages.

• Group Privacy: A corollary of differential privacy; if a user contributes k documents, the privacy guarantee degrades by a factor of k.
• Enforcement: Requires grouping all examples from a user and clipping the sum of their gradients as a single unit before adding noise.
• Real-World Impact: Prevents an attacker from inferring an individual's writing style or opinions by aggregating signals across all their contributions in the training set.

Differentially Private Stochastic Gradient Descent is the foundational algorithm for training deep learning models, including NLP transformers, with provable privacy. It operates by:

• Per-example gradient clipping: Bounding the L2 norm of each individual training sample's gradient to a fixed threshold C to limit sensitivity.
• Calibrated noise injection: Adding isotropic Gaussian noise to the aggregated, clipped gradients before the weight update.
• Privacy accounting: Tracking cumulative privacy loss (ε, δ) across training steps using a Moments Accountant to ensure the total budget is not exceeded.

A critical phenomenon that significantly strengthens privacy guarantees in DP-NLP training. By randomly sampling a batch of data from the full dataset before applying the DP mechanism, the effective privacy loss is reduced. The probability that any single record is included in a batch creates uncertainty, amplifying the privacy guarantee. This is why DP-SGD uses Poisson sampling or shuffled batches—without it, achieving tight ε values at scale would be computationally prohibitive.

Large language models are prone to unintentional memorization of rare or unique sequences from their training data. DP-NLP directly mitigates this by bounding the influence of any single document. Key dynamics:

• Extractable memorization: Attackers can query a non-private model to regurgitate verbatim training text (e.g., names, addresses).
• DP's role: The clipped, noisy gradients prevent the model from encoding exact sequences, forcing it to learn only statistical patterns shared across many users.
• Trade-off: High privacy (low ε) degrades performance on long-tail linguistic phenomena and rare entity recognition.

A stronger granularity of protection where all contributions from a single individual (e.g., all emails, reviews, or documents authored by one user) are treated as a single unit. In DP-NLP:

• Group privacy: The sensitivity calculation must account for the maximum number of records a user can contribute.
• Implementation: Requires user-level gradient clipping, where all gradients from a user's data are first aggregated and clipped as a single entity before noise is added.
• Use case: Essential for federated learning on mobile keyboards or email clients where a user contributes a corpus of text, not just a single sentence.

The practical application of DP-SGD to adapt pre-trained language models (like BERT or GPT) on sensitive domain text. Instead of training from scratch:

• Parameter-efficient approach: Often combined with LoRA or adapters to reduce the number of trainable parameters, which lowers the sensitivity and required noise.
• Public pre-training, private fine-tuning: Leverages a model pre-trained on public data, then applies DP only during the sensitive adaptation phase, preserving strong linguistic capabilities.
• Ghost clipping: A memory-efficient technique that computes per-example gradient norms without materializing all per-example gradients, critical for large transformer models.

The Private Aggregation of Teacher Ensembles framework provides an alternative to DP-SGD for NLP tasks. It works by:

• Partitioning data: Splitting the sensitive text corpus into disjoint subsets.
• Training teacher models: Each teacher is trained on its own partition without privacy constraints.
• Noisy voting: At inference, teachers vote on the output label, and Laplacian noise is added to the vote counts.
• Knowledge distillation: A student model is trained on public data labeled by the noisy teacher ensemble, enabling unlimited public queries without further privacy loss.