Fisher Information Matrix

The Fisher Information Matrix (FIM) is a positive semi-definite matrix that quantifies the amount of information a random observable variable carries about an unknown parameter vector upon which the probability of that variable depends. Formally, for a model with parameters θ and data distribution p(x|θ), the FIM F(θ) is defined as the expected covariance of the score function (the gradient of the log-likelihood):

F(θ) = E[∇ log p(x|θ) ∇ log p(x|θ)^T]

• Intuition: It measures the sensitivity of the model's output distribution to changes in its parameters. A high Fisher value for a parameter indicates that perturbing it significantly changes the model's predictions, signifying its importance.
• In continual learning, the diagonal of the FIM is used as a parameter importance measure, forming the basis for algorithms like Elastic Weight Consolidation (EWC).

Under specific regularity conditions, the Fisher Information Matrix is asymptotically equivalent to the expected Hessian of the negative log-likelihood. This establishes a direct link to the loss landscape's curvature.

• Relation: F(θ) ≈ E[H(θ)], where H is the Hessian matrix of the loss. This means the FIM approximates the local curvature of the Kullback-Leibler (KL) divergence between model distributions.
• Practical Implication: In deep learning, the full Hessian is computationally intractable. The FIM's diagonal provides a scalable, positive approximation of this curvature. In EWC, the Fisher diagonal F_i acts as a spring constant in the quadratic penalty (λ/2 * Σ_i F_i (θ_i - θ*_i)^2), anchoring important parameters (high curvature) close to their old values.

The FIM is fundamental to statistical estimation theory via the Cramér–Rao bound. This theorem states that the covariance of any unbiased estimator θ̂ is bounded below by the inverse of the Fisher Information Matrix:

Cov(θ̂) ≥ F(θ)^{-1}

• Interpretation: This establishes the FIM as a measure of the best possible precision (minimum variance) achievable when estimating the parameters. A "larger" FIM (in a positive-definite sense) implies potentially lower estimation error.
• Continual Learning Insight: Parameters with high Fisher information are those the model has precisely estimated for a given task. The Cramér–Rao bound provides a theoretical justification for protecting them: changing these well-estimated, high-precision parameters is statistically inefficient and highly disruptive to the model's established knowledge.

The Fisher Information Matrix is, by construction, positive semi-definite (PSD). This property is critical for its use as a valid metric tensor in information geometry.

• Mathematical Guarantee: For any vector v, v^T F(θ) v ≥ 0. This ensures the quadratic penalty used in EWC is always non-negative, providing a stable, convex-like regularization term.
• Information Geometry: The FIM defines a Riemannian metric on the statistical manifold of model distributions. The distance between two nearby distributions p(x|θ) and p(x|θ+dθ) is measured by dθ^T F(θ) dθ. This natural gradient direction, given by F(θ)^{-1}∇L, is the steepest descent direction on this manifold, accounting for the model's intrinsic parameter sensitivity.

Computing the exact, full FIM for large neural networks is infeasible. Continual learning algorithms employ key approximations:

• Diagonal Approximation: Only the diagonal elements F_ii are computed and stored, reducing complexity from O(N²) to O(N) for N parameters. This assumes parameter importances are independent, which is a pragmatic but limiting assumption.
• Empirical Estimate: The expectation E[⋅] is approximated by a Monte Carlo average over the training dataset D for the old task: F ≈ (1/|D|) Σ_{x in D} ∇ log p(x|θ) ∇ log p(x|θ)^T.
• Practical Calculation: For a discriminative model with a categorical output (e.g., classification), the gradient of the log-likelihood is simply the gradient of the negative log-loss for the true label. The squared gradients (or their running average) accumulated during training on a task provide the empirical diagonal Fisher estimate.

While foundational, the use of the FIM in continual learning has notable limitations that practitioners must address.

• Task-Interference Assumption: The diagonal FIM measures importance per-parameter, not per-task-interaction. It cannot capture which combinations of parameters are important, a limitation addressed by more advanced methods using block-diagonal or Kronecker-factored approximations.
• Quadratic Penalty Rigidity: The (θ - θ*)^2 penalty assumes a symmetric, local optimum. It may overly constrain parameters if the optimal solution for a new task lies far away, hindering forward transfer.
• Computational Overhead: Even the diagonal approximation requires a backward pass over the old task data to compute gradients, incurring cost. Storing a Fisher diagonal for each previous task scales linearly with the number of tasks.
• Dependence on Data and Model State: The Fisher estimate is valid only near the parameters θ* where it was computed. Its accuracy degrades as parameters move during new task training, a phenomenon known as the "plasticity-stability" trade-off inherent in quadratic penalties.

Elastic Weight Consolidation (EWC) is the primary algorithm that operationalizes the Fisher Information Matrix for continual learning. It adds a quadratic penalty term to the loss function, preventing important parameters (as measured by the diagonal of the FIM) from changing significantly when learning a new task.

• Mechanism: The penalty is scaled by the estimated Fisher information for each parameter, making the loss landscape "stiffer" in directions critical to old tasks.
• Connection to FIM: EWC approximates the posterior distribution over parameters given previous tasks, using the diagonal FIM as a measure of parameter importance. This transforms a theoretical statistical quantity into a practical regularization tool.

Synaptic Intelligence (SI) is an online, computationally efficient alternative to EWC that also estimates parameter importance for regularization. Instead of calculating the Fisher Information Matrix, SI tracks the cumulative weight change of each parameter throughout training, scaled by the loss improvement.

• Online Estimation: SI computes importance weights during the initial training of a task, requiring no separate post-hoc calculation like the FIM.
• Comparison to FIM: While FIM measures the sensitivity of the model's output distribution to parameter changes, SI measures the empirical contribution of each parameter to reducing the loss. Both result in a per-parameter importance score used for consolidation.

This is the overarching category of continual learning algorithms to which FIM-based techniques belong. These methods add a penalty term to the standard loss function to constrain updates to parameters deemed important for previous tasks.

• Core Principle: Protect knowledge by making the loss function sensitive to moving certain parameters.
• Key Variants:
  • FIM-based: EWC, Online EWC.
  • Path Integral-based: Synaptic Intelligence (SI).
  • Distillation-based: Learning without Forgetting (LwF), which uses a different form of regularization via output logits.
• Advantage: No need to store raw data, only task-specific metadata (e.g., importance matrices).

The Stability-Plasticity Dilemma is the fundamental theoretical challenge that the Fisher Information Matrix helps to address. It describes the trade-off between a neural network's ability to retain old knowledge (stability) and its capacity to integrate new information (plasticity).

• FIM's Role: The diagonal of the FIM quantifies this dilemma at the parameter level. A high Fisher value for a weight indicates it is crucial for stability (old tasks). A low value indicates it can be plastic for learning new tasks.
• Algorithmic Goal: Methods like EWC use the FIM to explicitly manage this trade-off, applying strong regularization to stable parameters while allowing plasticity in others.

Parameter Importance is the core concept estimated by the Fisher Information Matrix in continual learning. It is a per-parameter scalar value that quantifies how much a small change to that parameter affects the model's output distribution or loss on a given task.

• FIM Definition: For a parameter θᵢ, its importance is often taken as the diagonal entry Fᵢᵢ, which is the expected squared gradient of the log-likelihood with respect to θᵢ.
• Interpretation: High importance means the parameter is precisely tuned for the task; changing it will significantly degrade performance.
• Usage: This importance vector is the key metadata saved after learning each task and is used to constrain future learning.

Backward Transfer (BWT) is a critical evaluation metric for continual learning algorithms that use techniques like FIM-based regularization. It measures the impact of learning a new task on the performance of previously learned tasks.

• Calculation: Average change in accuracy on all previous tasks after learning a new task.
• Goal of FIM Methods: To achieve positive backward transfer (new learning improves old tasks) or, at minimum, minimize negative backward transfer (catastrophic forgetting).
• Connection: The effectiveness of the FIM importance estimation is directly reflected in the BWT metric. A well-estimated FIM should lead to BWT ≈ 0 or positive, indicating successful consolidation.