Extreme Value Theory (EVT)

The foundational EVT approach for OOD detection that models exceedances above a high threshold u using the Generalized Pareto Distribution (GPD). Instead of analyzing the entire activation distribution, POT focuses exclusively on the tail where extreme in-distribution scores reside.

• Threshold Selection: Uses mean residual life plots or stability plots to choose u in the tail region
• GPD Fit: Shapes parameters ξ and σ are estimated via maximum likelihood on excesses
• Calibration: Transforms raw softmax scores into calibrated probabilities of inclusion

An alternative EVT formulation that partitions the activation sequence into fixed-size blocks and models the distribution of the maximum value within each block using the Generalized Extreme Value (GEV) distribution.

• GEV Family: Unifies Fréchet (ξ > 0), Gumbel (ξ = 0), and Weibull (ξ < 0) distributions
• Block Size Trade-off: Larger blocks reduce bias but increase variance of parameter estimates
• Application: Useful when raw exceedance data is sparse or threshold selection is ambiguous

The specific EVT application popularized by OpenMax and related open-set classifiers. A Weibull distribution is fit to the tail of distances between correct class activations and their class mean vectors.

• Per-Class Modeling: Separate Weibull fits for each known class capture class-conditional extreme behavior
• Recalibration: Raw logits are discounted by the Weibull CDF probability of belonging to the tail
• Unknown Class Score: The residual probability mass is assigned to an explicit 'unknown' pseudo-class

A key diagnostic for validating EVT applicability in OOD detection. The estimated shape parameter ξ and modified scale parameter should remain approximately constant as the threshold u varies within the tail region.

• Stability Plots: Visual inspection of parameter estimates across threshold values
• Violation Implication: Non-stability suggests the tail has not yet entered the GPD domain of attraction
• Automated Selection: Algorithms can identify the lowest threshold where stability begins

Translates EVT tail models into operationally meaningful OOD thresholds. The return level associated with a probability p is the activation value expected to be exceeded with probability p on any given inference call.

• Quantile Mapping: Inverts the fitted GPD/GEV to obtain high-quantile estimates
• Confidence Bounds: Delta method or profile likelihood provides uncertainty intervals on return levels
• Practical Use: Sets rejection thresholds corresponding to desired false-positive rates (e.g., 1-in-10,000)

The estimated shape parameter ξ (tail index) directly quantifies the heaviness of the in-distribution activation tail, which correlates with OOD detection difficulty.

• ξ > 0 (Heavy Tail): Indicates extreme activations occur more frequently, making in-distribution and OOD scores harder to separate
• ξ ≤ 0 (Light/Bounded Tail): Suggests a natural ceiling on in-distribution scores, providing clearer separation from OOD inputs
• Diagnostic Use: Comparing ξ across model architectures or training regimes guides OOD robustness improvements

The GEV distribution is the limit distribution for normalized block maxima, unifying three families of tail behavior into a single parametric form governed by the shape parameter ξ.

• ξ = 0 (Gumbel): Light-tailed distributions like the normal, where extremes decay exponentially.
• ξ > 0 (Fréchet): Heavy-tailed distributions like the Cauchy, where extremes decay polynomially.
• ξ < 0 (Weibull): Bounded distributions with a finite upper endpoint.

In OOD detection, fitting a GEV to the maximum activation values per layer allows statistical calibration of what constitutes an extreme response.

The POT approach models exceedances above a high threshold u using the Generalized Pareto Distribution (GPD), offering more efficient use of extreme data than block maxima methods.

• Scale parameter σ: Controls the spread of exceedances.
• Shape parameter ξ: Determines tail heaviness, identical in interpretation to the GEV.
• Threshold selection: A critical bias-variance tradeoff—too low violates asymptotic theory, too high reduces sample size.

For OOD scoring, the fitted GPD provides a principled probability that a given activation magnitude belongs to the in-distribution tail.

OpenMax replaces the standard softmax layer with a Weibull-calibrated activation vector to explicitly model an 'unknown' class probability.

• Fits a Weibull distribution to the distance between correctly classified training samples and their class mean activation vectors.
• At inference, computes the CDF of each activation under its class Weibull model to recalibrate scores.
• The recalibrated scores are redistributed, with the residual mass assigned to the unknown pseudo-class.

This is a direct application of EVT to open-set recognition, using the Weibull as a special case of the GEV for bounded tails.

The Hill estimator is a classic method for estimating the tail index α = 1/ξ of heavy-tailed distributions, crucial for characterizing the extremal behavior of activation statistics.

• Computed from the k largest order statistics of a sample.
• A stable region in the Hill plot (Hill estimate vs. k) indicates a reliable tail index estimate.
• Heavy tails (α < 2) imply infinite variance, meaning extreme activations occur more frequently than Gaussian assumptions predict.

In deep network activations, heavy-tailed behavior has been empirically observed, motivating EVT-based thresholds over simple standard deviation rules.

A return level is the value expected to be exceeded on average once every m observations, providing an interpretable threshold for OOD rejection.

• Derived from the fitted GEV or GPD quantile function.
• A 1000-observation return level means the activation value is expected to be exceeded only once per 1000 in-distribution samples.
• Setting the OOD threshold at a high return level (e.g., 10,000-observation) provides a statistically principled false-positive rate.

This transforms abstract tail probabilities into operational thresholds that engineers can reason about directly.

EVT-based OOD detection primarily targets epistemic uncertainty—the model's lack of knowledge about unfamiliar inputs—by modeling when activations fall into extreme, untrained regimes.

• Aleatoric uncertainty: Irreducible noise in the data itself (sensor error, class overlap). EVT does not address this directly.
• Epistemic uncertainty: Reducible with more data. EVT quantifies the probability that a given activation is too extreme to be explained by the training distribution.
• Combining EVT thresholds with Monte Carlo Dropout or Deep Ensembles provides a more complete uncertainty decomposition.

This distinction is critical: EVT flags distributional novelty, not inherent data ambiguity.