Out-of-Distribution (OOD) Detection

Post-hoc methods apply a detection score to a pre-trained model's outputs without modifying the original training process. This is the most practical approach for production systems where retraining is expensive.

• Maximum Softmax Probability (MSP): The simplest baseline—uses the highest softmax value as a confidence proxy. It assumes OOD inputs trigger lower maximum probabilities.
• Energy-Based Score: Uses the Helmholtz free energy function E(x) = -T * log(sum(exp(f_i(x)/T))). In-distribution data maps to low energy; OOD data maps to high energy.
• ODIN: Enhances MSP by applying temperature scaling and small input perturbations to widen the gap between ID and OOD scores.
• GradNorm: Leverages the observation that the gradient magnitude of the KL divergence is typically higher for in-distribution samples.

Instead of analyzing output logits, these methods operate on the model's penultimate layer embeddings. The core assumption is that in-distribution features cluster tightly, while OOD features scatter.

• Mahalanobis Distance: Fits class-conditional Gaussian distributions to embeddings. The distance to the nearest class centroid, weighted by the covariance matrix, serves as the OOD score.
• KNN Distance: A non-parametric alternative. Computes the distance to the k-th nearest neighbor in the training set's feature space. Inputs far from the training manifold are rejected.
• Virtual Logit Matching (ViM): Projects features onto the residual space of principal components and combines this with the original logits, capturing information discarded by the softmax layer.
• Hyperspherical Embedding: Constrains features to a unit sphere, aligning class directions and preventing feature collapse, which improves separability.

The foundational baseline for OOD detection. MSP uses the highest softmax output as a confidence proxy, operating on the assumption that models produce lower probabilities for unfamiliar inputs. While computationally cheap, it suffers from overconfidence—modern neural networks often assign high softmax scores to OOD inputs, creating a narrow separation gap between known and unknown classes.

A probabilistic framework that maps inputs to scalar energy values using the Helmholtz free energy function. In-distribution data occupies low-energy basins, while OOD samples land in high-energy regions. Unlike softmax-based methods, the energy score is aligned with the input density, providing a more theoretically grounded separation criterion that doesn't require access to outlier data during training.

A two-stage post-hoc method that enhances MSP through temperature scaling and input perturbation. By raising the softmax temperature, ODIN smooths the probability distribution. It then adds a small gradient-based perturbation to the input, pushing it toward the nearest class boundary. This combination widens the gap between in-distribution and OOD softmax scores without retraining the model.

A parametric detection method that models each class as a Gaussian distribution in feature space. For a test input, it computes the Mahalanobis distance to the nearest class centroid, capturing the covariance structure of the training data. This approach leverages intermediate layer representations rather than final logits, often yielding more robust OOD signals by exploiting the geometry of learned feature manifolds.

A training-time intervention that leverages an auxiliary outlier dataset to teach the model explicit OOD heuristics. By exposing the network to diverse outliers during training and enforcing low confidence or uniform predictions on them, the model learns to generalize its detection capability to unseen OOD distributions. This method consistently outperforms post-hoc techniques when a representative outlier corpus is available.

A post-hoc method addressing activation overconfidence. ReAct observes that OOD inputs often trigger abnormally high activations in certain units. By clipping activations at a threshold derived from in-distribution statistics, it reduces the network's false confidence on OOD samples. This simple rectification step significantly improves OOD detection without modifying model architecture or requiring auxiliary data.