Typicality Test

A typicality test rejects inputs that fail to reside in the typical set—the high-probability region where most samples from the distribution actually fall. This addresses the density paradox where out-of-distribution (OOD) inputs can paradoxically receive higher likelihood scores than in-distribution data. The typical set is defined using information-theoretic principles, specifically the asymptotic equipartition property (AEP), which states that for large dimensions, almost all probability mass concentrates in a narrow shell of constant entropy.

• Key insight: High likelihood ≠ typicality
• Mathematical basis: Asymptotic equipartition property
• Dimension dependence: Becomes more pronounced in high-dimensional spaces

Standard likelihood-based OOD detection fails because generative models can assign high likelihood to nonsensical inputs that share superficial statistics with training data. Typicality tests correct this by evaluating whether a sample's negative log-likelihood falls within the expected entropy range of the true distribution. A sample is considered typical if its information content approximates the entropy of the source distribution.

• Failure mode: Constant background images score high likelihood
• Correction mechanism: Compare per-dimension entropy to expected values
• Implementation: Compute NLL and compare against distribution-specific thresholds

To distinguish between high-likelihood OOD inputs and genuine in-distribution samples, typicality tests often employ a background model or reference distribution. The likelihood regret metric subtracts the log-likelihood under a generic background model (trained on broad, diverse data) from the log-likelihood under the target model. This cancels out common high-probability features like smoothness or local pixel correlations that inflate raw likelihood scores.

• Likelihood regret formula: LR(x) = log p_target(x) - log p_background(x)
• Background model choice: Often a generic image compressor or simple density estimator
• Effect: Eliminates confounding factors unrelated to semantic content

A practical implementation of typicality testing leverages the running mean and variance stored in batch normalization layers. During inference, the distance between a test sample's feature statistics and the stored training statistics serves as a powerful typicality signal. Large deviations in channel-wise means or variances indicate the sample originates from a different distribution, even if the final softmax output appears confident.

• Detection signal: Mahalanobis distance in feature space
• Layer selection: Intermediate BN layers provide richer signals than final layers
• Computational cost: Negligible overhead—statistics are already computed during forward pass

Raw likelihood scores are biased toward low-complexity inputs—blank images, constant signals, or repetitive patterns often score higher than semantically meaningful data. Typicality tests apply complexity-aware corrections using minimum description length (MDL) principles or by normalizing likelihood by the input's intrinsic dimensionality. This prevents the model from accepting trivial adversarial examples that exploit the likelihood-complexity correlation.

• MDL principle: Prefer models that compress data efficiently
• Complexity metrics: Compression ratio, gradient magnitude, spectral entropy
• Adversarial defense: Rejects inputs designed to maximize likelihood through simplicity

Advanced typicality tests operate at multiple scales of representation, from raw pixels to high-level semantic features. A sample must demonstrate typicality across all scales to be accepted. This hierarchical approach catches OOD inputs that match coarse statistics but fail at fine-grained levels, or vice versa. Wavelet decompositions and feature pyramid networks provide natural multi-scale representations for this analysis.

• Scale levels: Pixel space → texture patches → semantic features
• Aggregation method: Product of per-scale typicality scores or logical AND
• Robustness: Prevents adversarial inputs that fool single-scale detectors

A metric that directly addresses a core failure mode of standard likelihood-based OOD detection: the tendency of generative models to assign high likelihood to nonsensical inputs. Likelihood Regret compares the likelihood of a test sample under a trained model to its likelihood under a background model, effectively canceling out the influence of input complexity. This is a direct alternative to the Typicality Test for correcting likelihood's blind spots.

A probabilistic framework that assigns low energy values to in-distribution data and high energy to OOD data. The Helmholtz free energy function serves as a discriminative score that is theoretically aligned with the input's probability density. Unlike raw likelihood, energy scores are less susceptible to the high-likelihood OOD paradox that the Typicality Test was designed to solve.

A classification framework requiring models to accurately classify known classes while simultaneously rejecting samples from unknown classes. The Typicality Test provides a statistical mechanism for this rejection decision. OpenMax, a canonical algorithm in this space, replaces the standard softmax layer with a Weibull-calibrated activation vector that explicitly models the probability of an 'unknown' class.

A generative modeling approach that transforms a simple base distribution into a complex one through a sequence of invertible mappings, enabling exact likelihood computation. Normalizing Flows are a natural testbed for Typicality Tests because they provide tractable densities. Research has shown that flows can still assign high likelihood to OOD inputs, making typicality-based rejection a critical complementary technique.

A parametric OOD detection method that computes the distance of a feature representation to the nearest class-conditional Gaussian distribution. By capturing the covariance structure of the training data, this score implicitly tests whether an input resides in the high-density region of the feature space—a concept closely related to the Typicality Test's assessment of whether a sample belongs to the typical set rather than low-probability modes.