Out-of-Distribution Detection

A baseline method where the model's confidence score is used as a proxy for in-distribution likelihood. Inputs are flagged as OOD if the maximum predicted probability (softmax score) for any class falls below a defined threshold.

• Mechanism: Leverages the observation that models tend to be overconfident and produce high softmax scores for in-distribution data, even when incorrect.
• Limitation: Prone to failure with overconfident miscalibrated models and certain adversarial examples.
• Example: An image classifier trained on cats and dogs might assign a low max softmax probability (e.g., 0.3) to an image of a car, signaling a potential OOD sample.

These techniques measure the statistical distance or similarity between a test input's representation and the known in-distribution data within a model's latent space.

• Mahalanobis Distance: Calculates the distance of a test sample's feature vector from the class-conditional Gaussian distributions fitted to training data features. It is effective for detecting both semantic and covariate shift.
• k-Nearest Neighbors (k-NN): Uses the distance to the k-nearest training embeddings in the feature space; larger distances indicate OOD samples.
• Key Insight: Assumes in-distribution samples form compact clusters in the feature space learned by the model's penultimate layer.

This approach involves explicitly modeling the probability distribution of the training data, then evaluating the likelihood of a new input under this model.

• Models Used: Normalizing Flows, Variational Autoencoders (VAEs), or Gaussian Mixture Models (GMMs) are trained to estimate p(x), the probability density of in-distribution inputs.
• Detection: Inputs with a computed likelihood p(x) below a threshold are classified as OOD.
• Challenge: Can be misled by background statistics or simple features not relevant to the core task, sometimes assigning higher likelihood to OOD data.

A training-time technique that improves OOD detection by exposing the model to auxiliary OOD data during training, teaching it to explicitly lower confidence for anomalous inputs.

• Process: The model is trained on the primary dataset with an added objective to uniformly distribute softmax probabilities for examples from a diverse, held-out 'outlier' dataset.
• Benefit: Converts OOD detection from an unsupervised to a supervised problem, leading to more robust detectors.
• Auxiliary Data: Often uses large, generic datasets (e.g., 80 Million Tiny Images) that are disjoint from the primary training and test sets.

These methods analyze the model's gradients or backward pass in response to an input, based on the hypothesis that OOD data induces different gradient signals than in-distribution data.

• Gradient Magnitude: The norm of the gradients with respect to the input or model parameters can be larger for OOD samples as the model struggles to classify them.
• Spectral Analysis: Techniques like Spectral Normalized Gaussian Process (SNGP) layers enhance a model's ability to provide uncertainty estimates by analyzing the spectrum of the last layer's weights, improving OOD detection.
• Use Case: Particularly useful in conjunction with Bayesian neural network approximations.

Leveraging multiple models or specialized modules to capture epistemic uncertainty, which is high for OOD inputs.

• Deep Ensembles: Train multiple models with different random initializations. Disagreement or variance in their predictions signals OOD data.
• Monte Carlo Dropout: Treats dropout at inference time as an approximation of a Bayesian neural network. Variance across stochastic forward passes indicates uncertainty and potential OOD status.
• Detector Heads: Attach a separate, simple binary classifier (e.g., a logistic regression or small MLP) to the model's feature embeddings, trained specifically to distinguish ID vs. OOD using a held-out validation set.

The process of identifying when an LLM generates factually incorrect or nonsensical information not grounded in its training data or provided context. While OOD detection flags unfamiliar inputs, hallucination detection flags erroneous outputs. Key methods include:

• Self-consistency checks (sampling multiple outputs)
• Fact verification against knowledge bases
• Entropy-based scoring of token probabilities
• Contradiction detection within the generated text itself.

A model's resistance to producing incorrect or unsafe outputs when presented with intentionally crafted, malicious inputs designed to fool it. This is closely related to OOD detection, as adversarial examples are often statistical outliers. Techniques to improve robustness include:

• Adversarial training with perturbed inputs
• Gradient masking to obscure attack surfaces
• Input reconstruction to filter noise
• Ensemble methods to average over model uncertainties.

The process of checking whether an LLM's output is substantiated by and correctly references the source material or context provided to it, such as in a Retrieval-Augmented Generation (RAG) system. It ensures outputs are not fabrications. This differs from OOD detection but is complementary; a well-grounded system should still refuse to answer if the query itself is OOD. Verification methods include:

• Citation accuracy checks
• Claim-attribution matching
• Semantic similarity scoring between output and source chunks.

An ensemble moderation technique where multiple specialized ML classifiers are applied sequentially or in parallel to validate an LLM output. An OOD detector often acts as the first classifier in this chain, triaging queries before they reach more expensive content-specific models (e.g., for toxicity, bias, PII). A typical chain might be:

1. OOD Detector: Is this query in-domain?
2. Intent Classifier: What is the user asking for?
3. Safety Moderation: Is the output toxic/harmful?
4. Fact-Checker: Is the output accurate?

A model's trained behavior to decline to generate outputs for requests that are harmful, unethical, illegal, or outside its operational boundaries. OOD detection systems often trigger a refusal. There are two primary types:

• Hard-coded refusals: A predefined safe response (e.g., "I cannot answer that.") is returned.
• Model-internal refusals: The model itself is fine-tuned (e.g., via RLHF) to generate a refusal token or statement. Effective refusal requires clear communication to the user without being evasive.

A standardized dataset and evaluation protocol used to measure and compare the safety and robustness of different language models. OOD detection capability is a key metric within these benchmarks. Prominent examples include:

• TruthfulQA: Measures a model's tendency to generate falsehoods.
• ToxiGen: A large-scale benchmark for detecting hate speech.
• HELM (Holistic Evaluation of Language Models): Includes robustness to distribution shift.
• Big-Bench Hard: Contains tricky, out-of-domain style tasks. These benchmarks provide quantitative scores for model comparison.