Out-of-Distribution (OOD) Detection

A foundational approach where the model learns the probability density function of the training data. Inputs falling in low-likelihood regions are flagged as OOD.

• Explicit Models: Normalizing flows and PixelCNNs directly compute log-likelihood.
• Paradox: High-likelihood OOD samples can occur; a constant noise image may score higher than an in-distribution image, necessitating likelihood ratio methods.
• Typicality Test: Evaluates if a sample resides in the typical set of the distribution, not just high-density areas.

Leverages the penultimate layer embeddings of a pre-trained classifier. The core assumption is that in-distribution samples cluster tightly, while OOD inputs map to distant, diffuse regions.

• Mahalanobis Distance: Computes the distance from class-conditional Gaussian centroids, accounting for feature covariance.
• K-Nearest Neighbors (KNN): Flags samples whose distance to their k-th nearest training embedding exceeds a threshold.
• Advantage: Does not require retraining the classifier; operates as a post-hoc method.

Uses the Helmholtz free energy function derived from a discriminative classifier's logits. The energy score is theoretically aligned with the input's probability density and is less susceptible to the overconfidence of Softmax.

• Formula: E(x) = -T * log Σ exp(f_i(x)/T), where T is a temperature parameter.
• Property: In-distribution samples exhibit lower, more negative energy values.
• Unified Framework: Connects discriminative classifiers to generative models, providing a principled score for OOD detection without explicit density modeling.

Analyzes the gradient information of a neural network with respect to its parameters or inputs. OOD samples induce distinct gradient patterns compared to in-distribution data.

• GradNorm: Measures the L1 norm of the KL divergence gradient between the Softmax output and a uniform distribution. OOD inputs force a larger parameter update.
• Input Gradients: The magnitude of the loss gradient with respect to the input pixels is typically higher for OOD samples.
• Mechanism: Exploits the fact that the model is in a flatter loss landscape minimum for training data.

Post-hoc modification of the classifier's output layer to recalibrate confidence. Standard Softmax probabilities are often overconfident for OOD inputs; scaling the logits mitigates this.

• Temperature Scaling: Divides logits by a scalar T > 1 to soften the probability distribution.
• LogitNorm: Constrains the norm of the logit vector during training, decoupling magnitude from confidence and forcing the model to rely on angular separation.
• Impact: Directly addresses the core failure mode where OOD inputs produce high maximum Softmax probability.

Applies a transformation to the input to amplify the signal difference between in-distribution and OOD data before scoring.

• ODIN: Adds a small, calibrated gradient-based perturbation to the input and applies temperature scaling. The perturbation pushes in-distribution samples toward higher confidence faster than OOD samples.
• Input Corruption: Tests the stability of predictions under noise; OOD predictions collapse more rapidly.
• Rationale: In-distribution data resides on a learned manifold and is more resilient to small adversarial adjustments.

While often conflated, these tasks have a critical distinction in scope and assumptions:

• Anomaly Detection: Identifies rare or atypical samples within the training distribution, often using unsupervised methods like autoencoders or one-class SVMs
• OOD Detection: Identifies samples from a semantically different distribution entirely, requiring the model to recognize fundamental shifts in data generating processes
• Practical implication: A fraudulent transaction (anomaly) still belongs to the financial domain; a cat photo shown to a medical imaging model (OOD) represents a complete distributional shift