Influence Functions

The core mechanism estimates how the model's optimal parameters would change if a specific training example were upweighted by an infinitesimal epsilon. This is computed by measuring the alignment between the gradient of the loss on the training point and the inverse Hessian-vector product with the test loss gradient.

• Avoids the prohibitive cost of leave-one-out retraining.
• Identifies prototypical vs. outlier training examples.
• A large positive influence indicates removing the point would improve the test loss.

Influence functions can estimate how a prediction would change if a training input were locally perturbed. This reveals which features in the training data the model is sensitive to, enabling fine-grained data debugging.

• Computed via the gradient of the influence with respect to the training input.
• Helps uncover spurious correlations learned from flawed training examples.
• Useful for identifying label errors that disproportionately skew decision boundaries.

The computational bottleneck is forming the inverse Hessian-vector product. Since the full Hessian is intractable for modern deep networks, efficient approximation is critical.

• Conjugate gradient optimization iteratively approximates the HVP without materializing the Hessian.
• LiSSA (Linear time Stochastic Second-Order Algorithm) uses a Neumann series expansion for a stochastic estimate.
• Modern implementations leverage automatic differentiation and per-sample gradients for exact computation on small models.

The framework distinguishes between two related quantities: the influence on the model parameters and the influence on the test loss.

• Parameter influence: The actual change in the weight vector if a point were removed.
• Loss influence: The resulting change in the prediction error on a specific test point, derived by chaining the parameter influence with the test gradient.
• The loss influence is the primary metric for identifying harmful training examples that degrade specific test predictions.

A training point's self-influence measures its impact on its own prediction. High self-influence points are often atypical, mislabeled, or conflicting with the broader data distribution.

• Used as a ranking criterion for data cleaning.
• Removing high self-influence examples can improve generalization and robustness.
• Contrasts with memorization, where a point has high self-influence but negligible impact on other test points.

Influence functions are a direct generalization of Cook's distance from classical linear regression to non-convex deep learning models. In linear models, the formula is exact; in neural networks, it relies on a quadratic approximation of the loss around the optimal parameters.

• The approximation quality degrades for large parameter changes or highly non-convex loss landscapes.
• Works best for models trained to convergence where the gradient is near zero.
• Provides a local, first-order approximation of a global counterfactual.