Influence Functions

The core mechanism involves asking a counterfactual question: if a specific training point z were upweighted by an infinitesimal amount ε during training, how would the model's loss on a test point z_test change? This is computed using the Influence Function formula: I(z, z_test) = -∇_θ L(z_test, θ̂)ᵀ H_θ̂⁻¹ ∇_θ L(z, θ̂). This requires the Hessian matrix of the training loss, representing the model's curvature around the optimal parameters.

A primary application is efficiently approximating the effect of Leave-One-Out (LOO) retraining without the prohibitive cost of retraining the model n times. By setting the weight ε to -1/n, the influence function estimates how the model parameters and predictions would change if a specific data point were completely removed from the training set. This is critical for identifying mislabeled examples or outliers that disproportionately degrade model performance.

Directly inverting the Hessian H_θ̂⁻¹ is computationally intractable for modern deep networks with millions of parameters. The practical breakthrough comes from using implicit Hessian-vector products (HVPs) via stochastic estimation techniques like LiSSA (Linear time Stochastic Second-Order Algorithm) or conjugate gradients. These methods approximate H⁻¹v without materializing the full matrix, reducing the complexity from O(p³) to roughly the cost of a gradient computation.

Influence functions can pinpoint poisoning attacks and naturally occurring adversarial examples in the training set. By computing the influence of every training point on a misclassified test example, one can identify the small subset of training data that most strongly 'pushed' the decision boundary in the wrong direction. Removing these high-negative-influence points often restores correct classification, providing a powerful tool for data debugging and model patching.

The classical influence function derivation assumes the empirical risk minimizer θ̂ is a global minimum and that the loss is strictly convex and twice-differentiable. In deep learning, these assumptions are violated due to non-convex loss landscapes and discontinuous architectures (e.g., ReLU, dropout). This can cause the first-order Taylor approximation to break down, leading to inaccurate influence estimates, especially for large parameter changes or when the Hessian has negative eigenvalues.

The self-influence of a training point—its influence on its own prediction—reveals how memorized or atypical a sample is. A high positive self-influence score indicates the model relies heavily on that specific point to predict itself correctly, a hallmark of memorization rather than generalization. Conversely, points with low self-influence are well-explained by the broader data distribution. This metric is a powerful lens for understanding training dynamics and detecting label noise.

The measurement of a model's reducible uncertainty stemming from a lack of knowledge or limited data. Influence functions provide a direct link to this concept.

• Relationship: A test point with high epistemic uncertainty is one whose prediction is heavily influenced by a small, specific subset of training data. Removing that data would drastically change the prediction.
• Techniques: Monte Carlo Dropout and Deep Ensembles approximate Bayesian inference to estimate uncertainty, but influence functions offer a more targeted, data-centric explanation.
• Practical Use: Identifying regions of the input space where collecting more training data would most improve model reliability.