Machine Unlearning

The gold standard for verifiable deletion. The training dataset is divided into isolated, non-overlapping shards, each training an independent constituent model. To unlearn a data point, only the specific shard containing that point is retrained from scratch.

• SISA Training: The primary framework (Sharded, Isolated, Sliced, Aggregated).
• Verifiability: Provides a mathematical proof of removal, not just a probabilistic guarantee.
• Trade-off: Incurs a storage overhead for maintaining multiple models and a latency spike during the shard retraining process.

Instead of retraining, this method directly manipulates the model's weights to reverse the gradient updates induced by the target data. It leverages the Newton step to compute a weight perturbation that approximates the model's state as if the data was never seen.

• Influence Functions: Used to estimate the impact of a single training point on the final weights.
• Certified Removal: Bounds the statistical distance between the scrubbed model and a perfectly retrained model.
• Scalability: Much faster than exact unlearning but can accumulate approximation errors over sequential deletion requests.

A model trained with a strict privacy budget (ε) inherently limits the influence of any single record. In this framework, unlearning is a byproduct of the training process itself. If the model is differentially private, the removal of a single record has a mathematically bounded, negligible impact on the model's output distribution.

• No Retroactive Action: No computation is required at deletion time.
• Utility Gap: The noise injected by DP-SGD often degrades accuracy on complex tasks.
• Composability: The privacy guarantee degrades predictably with sequential queries, aligning with data retention policies.

A 'teacher' model is trained on the full dataset. To unlearn a subset, the teacher's knowledge is distilled into a 'student' model, but the student is explicitly penalized for matching the teacher's output on the forget set. The student learns to mimic the teacher on the retain set while maximizing entropy on the deleted data.

• Architecture Flexibility: Works without access to the original training pipeline.
• Catastrophic Forgetting: The objective is to induce controlled forgetting on specific data points.
• Utility Preservation: The student often maintains competitive accuracy on the retain set.

Leverages Trusted Execution Environments (TEEs) to create a cryptographic boundary around the training process. The model is encrypted at rest. Unlearning is executed by destroying the decryption key for the specific data shard or model checkpoint, rendering the data influence cryptographically inaccessible.

• Physical Security: Relies on hardware root of trust rather than algorithmic complexity.
• Instant Revocation: Deletion is equivalent to secure key destruction.
• Infrastructure Dependency: Requires specific CPU architectures (e.g., Intel SGX, AMD SEV).

Rather than shrinking the model's knowledge, this technique expands the decision boundary to exclude the forgotten data. It fine-tunes the model to maximize the loss on the forget set while minimizing loss on the retain set, effectively pushing the forgotten samples across the classification boundary into a region of high uncertainty.

• Gradient Ascent: Applies inverse optimization steps on the target data.
• Imperfect Scrubbing: May leave residual traces of the data in the model's latent space.
• Speed: Extremely fast, often requiring only a few epochs of fine-tuning.

Sharded, Isolated, Sliced, Aggregated training is the foundational exact unlearning framework. Data is partitioned into independent shards, each training a constituent model. When a deletion request arrives, only the shard containing the target data is retrained, reducing computational cost from O(N) to O(N/S) where S is the number of shards.

• Isolation: No data point influences more than one shard
• Checkpointing: Intermediate model states are saved to accelerate retraining
• Aggregation: Final prediction is a simple ensemble of shard outputs

The primary threat that machine unlearning defends against. An MIA determines whether a specific record was in the training set by analyzing prediction confidence, loss values, or label outputs. Unlearning aims to render a deleted record indistinguishable from a non-member.

• Shadow models are trained to mimic target behavior and generate labeled attack data
• Black-box attacks require only API access to confidence scores or hard labels
• Metric: Attack AUC quantifies how well an adversary can distinguish members from non-members

A mathematical framework providing provable privacy guarantees by bounding the influence of any single record on model output. When combined with unlearning, DP offers a statistical alternative to exact deletion: if ε is small enough, the model's behavior is nearly indistinguishable whether or not a record was ever included.

• ε (epsilon): The privacy budget controlling leakage magnitude
• DP-SGD: Injects calibrated Gaussian noise during training
• Composition: Multiple DP mechanisms consume the budget additively

A per-instance metric that quantifies the likelihood a specific training record can be successfully identified by a membership inference attack. Risk scores enable targeted unlearning by identifying high-risk records for prioritized deletion.

• Calibrated: Scores reflect empirical attack success probability
• Granular: Computed per-example rather than dataset-wide
• Actionable: High-risk records trigger selective retraining or scrubbing

A model compression technique where a student model learns from the softened probability outputs of a teacher ensemble. When teachers are trained on disjoint data partitions, distillation can incidentally reduce membership information leakage in the student.

• Temperature scaling smooths teacher logits to transfer dark knowledge
• PATE: Extends this with noisy voting for formal DP guarantees
• Unlearning benefit: Student never directly accesses raw training data

A critical preprocessing step that bounds the L2 norm of per-sample gradients before noise addition in DP-SGD. Clipping limits the maximum influence any single record can exert on a model update, directly enabling both differential privacy and bounded unlearning sensitivity.

• Clipping threshold C: Maximum allowed gradient norm per sample
• Trade-off: Lower C improves privacy but may discard useful signal
• Unlearning connection: Bounded influence simplifies certified removal guarantees