Influence Function

Influence functions efficiently estimate the effect of retraining a model from scratch without a specific data point, without actually performing the costly retraining. By leveraging the Hessian matrix of the loss function, the method computes a closed-form approximation of the parameter shift. This transforms an O(n) retraining problem into a single gradient calculation, making it feasible to audit massive datasets for harmful outliers.

Beyond binary removal, influence functions quantify the impact of infinitesimally upweighting a training sample. The mathematical formulation computes the derivative of the optimal parameters with respect to a small increase in the sample's weight. This reveals which data points exert the most leverage on the model's decision boundary, identifying not just poisoned samples but also highly influential benign outliers that disproportionately shape model behavior.

The computational backbone relies on the Hessian matrix (second-order derivatives of the loss) and its inverse. Key implementation details include:

• Conjugate gradient methods avoid explicit O(p²) Hessian storage
• Stochastic estimation approximates the inverse Hessian-vector product
• LiSSA algorithm enables linear-time stochastic approximation This makes influence computation tractable for modern deep networks with millions of parameters.

Influence functions predict how the loss on a specific test point would change if a training point were removed. A positive influence score indicates the training point helped reduce test loss; a negative score signals harmful impact. For poisoned data, the influence on clean test examples is strongly negative, creating a detectable signature. This enables targeted data sanitization by ranking and removing the most negatively influential samples.

At its core, the influence of a training point z on a test point z_test is proportional to the inner product of their loss gradients, weighted by the inverse Hessian. This reveals a geometric interpretation: training points with gradient directions aligned to the test point's gradient are beneficial, while those with opposing gradient directions are detrimental. This alignment metric serves as a fast, interpretable screening tool for potential poisoning.

Influence functions exhibit strong capability in identifying mislabeled training examples. Samples with flipped or incorrect labels typically exert a disproportionately negative influence on correctly labeled test examples from the same class. By surfacing these high-negative-influence outliers, practitioners can automatically flag annotation errors and label-flipping attacks without requiring ground-truth labels for the training set.

The defensive process of filtering, transforming, or removing suspicious training samples before model training begins. Influence functions provide the mathematical justification for which points to remove.

• Anomaly thresholding: Flagging points with outlier influence scores
• Defensive distillation: Training a second model on sanitized outputs
• Proactive cleansing: Removing high-influence poisoned samples identified during exploratory analysis

A defense method that identifies poisoned data by analyzing the singular value decomposition of feature representations. The latent separability of corrupted samples from clean ones becomes visible in the spectrum.

• Right singular vectors reveal directions of maximum variance between clean and poisoned populations
• Correlation with influence functions: High-influence poisoned points often exhibit strong spectral signatures
• Centered kernel alignment measures similarity between the spectral signature and candidate anomalies

Assigns a numerical score to each data point based on its deviation from the expected distribution. Influence functions elevate anomaly scoring by measuring not just statistical oddness, but actual impact on model parameters.

• Mahalanobis distance: Measures multivariate distance from the distribution centroid
• Isolation forests: Tree-based anomaly detection that isolates outliers in few splits
• Influence-weighted scoring: Combining statistical anomaly scores with influence magnitude for prioritized review

The documented chronology of a dataset's origin, transformations, and chain of custody. When influence functions flag a sample as harmful, provenance enables tracing it back to its source for root cause analysis.

• Cryptographic hashing creates immutable fingerprints for each data version
• Lineage graphs map the full transformation pipeline from ingestion to training
• Attribution metadata links each sample to its collector, timestamp, and preprocessing steps

A defensive technique that caps the magnitude of individual gradients during training to prevent maliciously large updates from dominating the learning process. Complements influence-based detection by limiting the damage a single poisoned point can inflict.

• L2 norm clipping constrains the Euclidean norm of each gradient vector
• Adaptive clipping adjusts thresholds based on gradient distribution statistics
• Relationship to influence: Clipping reduces the maximum possible influence any single point can exert on the final parameters