Out-of-Distribution Detection

OOD detectors often model the probability density of the training data's latent representations. Inputs falling into low-density regions of this feature space are flagged as anomalous.

• Gaussian Mixture Models (GMMs) fit a multi-modal distribution to class-conditional features.
• Normalizing Flows learn an exact likelihood by transforming a simple base distribution into a complex one.
• Kernel Density Estimation (KDE) provides a non-parametric approach, though it scales poorly with high-dimensional embeddings.
• A threshold on the log-likelihood score separates in-distribution from OOD samples.

This approach computes the distance between a test sample's embedding and the training data manifold. Large distances indicate semantic dissimilarity.

• Mahalanobis Distance measures the number of standard deviations from the class-conditional mean, accounting for feature covariance.
• Cosine Similarity assesses angular distance in the hyperspherical latent space, often more robust to magnitude variations.
• k-Nearest Neighbors (k-NN) distance uses the average distance to the k closest training points as an anomaly score.
• These methods are simple to implement but sensitive to the quality of the learned representations.

EBMs assign a scalar energy value to an input configuration, where in-distribution data is assigned low energy and OOD data high energy.

• The Helmholtz Free Energy of a discriminative classifier aligns the logits to form an energy function.
• OOD detection is performed by thresholding the energy score without requiring an explicit generative model.
• This method leverages the smoothness of the logit space, often outperforming softmax confidence scores.
• EBMs provide a principled bridge between discriminative and generative approaches.

The magnitude and direction of gradients with respect to model parameters can signal whether an input is familiar to the network.

• Gradient Norm: OOD inputs often produce larger gradient norms as the network struggles to reconcile them with learned weights.
• GradNorm leverages the KL divergence between the softmax output and a uniform distribution, weighted by gradient magnitude.
• Input Preprocessing: Techniques like ODIN perturb inputs to increase the softmax temperature gap between in-distribution and OOD samples.
• These methods require a backward pass, adding computational overhead at inference time.

Raw softmax probabilities are notoriously miscalibrated for OOD detection. Normalizing the logit space provides a more reliable uncertainty signal.

• Maximum Softmax Probability (MSP) is the baseline but fails on high-confidence OOD inputs.
• LogitNorm applies L2 normalization to the logit vector, decoupling its magnitude from the prediction confidence.
• Temperature Scaling sharpens the softmax distribution, but a fixed temperature is often suboptimal for OOD separation.
• ReAct clips extremely high activations in the penultimate layer to prevent feature over-generalization.

Rather than relying on post-hoc scores, these methods explicitly train the model to recognize and reject OOD samples.

• Outlier Exposure (OE) introduces a large, diverse auxiliary dataset of outliers during training, forcing the model to learn a decision boundary with low uniform confidence on OOD inputs.
• Energy Regularization fine-tunes the model to assign high energy to outlier data while minimizing energy on in-distribution data.
• Contrastive Learning pulls in-distribution representations together while pushing OOD representations apart in the latent space.
• This approach requires careful selection of the auxiliary outlier dataset to avoid biasing the model.

The fundamental decomposition of predictive uncertainty. Epistemic uncertainty is the model's ignorance due to lack of data—it is reducible and high in OOD regions. Aleatoric uncertainty is the inherent, irreducible noise in the data. OOD detection primarily targets high epistemic uncertainty.

• Epistemic: High in regions far from training data
• Aleatoric: High in noisy or ambiguous in-distribution samples
• Key Insight: A robust OOD detector must disentangle these two sources to avoid flagging noisy but in-distribution data.

Leveraging the disagreement among multiple independently trained models to detect OOD inputs. Ensemble disagreement serves as a strong proxy for epistemic uncertainty.

• Train M models with different random seeds
• Measure the variance or mutual information of their predictive distributions
• High variance indicates the input is in an unexplored region of the input space
• More robust than single-model methods but computationally expensive at both training and inference time

A non-parametric Bayesian model that defines a distribution over functions. A GP's predictive variance naturally inflates in regions far from the training data, making it an intrinsic OOD detector.

• Defined by a kernel function (e.g., RBF) that encodes smoothness assumptions
• Provides calibrated uncertainty estimates without post-hoc adjustments
• Computationally expensive for large datasets (O(N³))
• Serves as a gold-standard baseline for uncertainty quality