Fisher Information Matrix

The FIM measures how much a small perturbation in a specific model weight changes the model's output distribution. In approximate unlearning, this identifies which parameters are most responsible for encoding a target data point. By calculating the empirical Fisher on the forget set, algorithms can apply a targeted penalty to high-sensitivity weights, erasing the data's influence while minimizing collateral damage to the model's general knowledge.

Standard gradient descent moves parameters in the steepest direction of the loss landscape, but this direction is not invariant to parameterization. The FIM enables natural gradient descent, which accounts for the underlying geometry of the probability distribution. In unlearning, this allows for more efficient and stable parameter updates when erasing data, ensuring that the removal step respects the true information geometry rather than an arbitrary Euclidean distance in weight space.

The FIM is the foundational component for computing influence functions, which estimate the effect of removing a training point without retraining. The influence of a data point is calculated using the inverse FIM multiplied by the gradient of the loss on that point. This provides a first-order approximation of how much each parameter would change if the model were retrained from scratch, enabling efficient certified removal audits.

In the context of data sharding and SISA training, the FIM helps determine the optimal way to partition data to minimize future unlearning costs. By maximizing the determinant of the FIM (D-optimality) for each shard, engineers can ensure that each subset is maximally informative, reducing the number of parameters that need to be retrained when a deletion request targets a specific shard.

The FIM serves as the precision matrix in a Laplace approximation of the posterior distribution over model parameters. In Bayesian unlearning frameworks, the posterior is approximated as a Gaussian centered at the maximum a posteriori estimate with covariance equal to the inverse FIM. Removing data then involves a rank-one update to this Gaussian, providing a principled probabilistic method for amnesiac unlearning with uncertainty quantification.

Elastic Weight Consolidation uses the FIM as a measure of synaptic importance to prevent catastrophic forgetting. When learning a new task or, inversely, when unlearning old data, the FIM computed on the retained data acts as a quadratic penalty. This anchors critical weights in place, ensuring that the removal of specific data points does not cascade into a broader degradation of model performance on the validation set.

A statistical tool that quantifies how removing or upweighting a single training point affects model parameters. Influence functions use the inverse of the Fisher Information Matrix to approximate parameter changes without retraining.

• Computes the effect of data removal on the loss landscape
• Essential for identifying high-impact training samples
• Provides the theoretical basis for efficient approximate unlearning

The square matrix of second-order partial derivatives of a scalar-valued loss function. The Fisher Information Matrix is equivalent to the negative expected Hessian of the log-likelihood under certain regularity conditions.

• Describes the local curvature of the loss landscape
• Used in Newton's method for optimization
• Computationally expensive to compute and invert for large models

An optimization method that uses the Fisher Information Matrix to precondition the gradient, accounting for the geometry of the parameter space. This enables more efficient learning steps that are invariant to parameter reparameterization.

• Moves in the steepest direction in distribution space, not parameter space
• Particularly effective for probabilistic models
• Requires computing or approximating the Fisher matrix inverse

A fundamental theorem stating that the variance of any unbiased estimator is bounded below by the inverse of the Fisher Information Matrix. This establishes the Fisher matrix as the definitive measure of how precisely a parameter can be estimated from data.

• Defines the theoretical limit of estimation accuracy
• More information = lower achievable variance
• Used to evaluate estimator efficiency in statistical modeling

A mathematical framework that provides provable privacy guarantees by injecting calibrated noise. The Fisher Information Matrix quantifies the sensitivity of model parameters to individual data points, directly informing the epsilon budget required for certified removal.

• Fisher information bounds the privacy loss from data inclusion
• Enables formal guarantees for machine unlearning procedures
• Connects parameter importance to provable privacy thresholds

A continual learning algorithm that uses the Fisher Information Matrix to identify parameters critical for previous tasks. It applies a quadratic penalty to slow learning on these important weights, preventing catastrophic forgetting.

• Fisher diagonal approximates per-parameter importance
• Enables sequential task learning without degradation
• Directly applicable to controlled, targeted unlearning scenarios