ReAct

ReAct identifies that extremely high activation values in the penultimate layer are a primary source of overconfidence for OOD inputs. It applies a simple, computationally cheap rectification by clipping activations above a threshold c to c. This threshold is typically calibrated on a held-out validation set of in-distribution data. The operation is defined as ReAct(x) = min(x, c), where c is the layer-specific clipping threshold.

A defining advantage of ReAct is its post-hoc nature. It does not require retraining, fine-tuning, or modifying the original model architecture. The clipping operation is applied directly to the activations during inference, making it immediately deployable on existing production models without access to the original training pipeline or data. This contrasts sharply with methods like Outlier Exposure that require auxiliary data and retraining.

ReAct is often combined with the Energy-Based Model (EBM) scoring function for state-of-the-art OOD detection. The EBM score uses the Helmholtz free energy: E(x) = -T * log(sum(exp(f_i(x)/T))). ReAct's activation clipping reduces the extreme logit values that inflate the energy score for OOD inputs, significantly widening the separation gap between in-distribution and out-of-distribution energy distributions.

The rectification is applied to the feature activations (the output of the global average pooling layer in CNNs or the penultimate layer in transformers), not the final logits. This is a crucial distinction. By normalizing the feature vector before it enters the final linear classifier, ReAct prevents the propagation of spuriously high unit activations that lead to high-confidence, incorrect predictions on semantically novel inputs.

The computational cost of ReAct is negligible. It consists of a single element-wise comparison and clamp operation per activation unit. This adds sub-millisecond latency to the inference pipeline, making it suitable for real-time, high-throughput production systems. It requires no additional forward or backward passes, unlike Monte Carlo Dropout or gradient-based methods such as GradNorm.

ReAct is motivated by the observation that neural networks with ReLU activations can produce unboundedly high activation magnitudes for inputs far from the training manifold. These high activations dominate the final logit computation, leading to artificially high softmax confidence. By capping these activations, ReAct forces the model to rely on the distributed pattern of activations learned from in-distribution data, which is disrupted for OOD inputs.

The foundational baseline for OOD detection that uses the highest softmax output as a confidence score. ReAct directly addresses MSP's core failure mode: overconfident predictions on OOD inputs.

• Mechanism: Extracts max(softmax(logits)) and rejects inputs below a threshold
• Weakness: Modern networks produce arbitrarily high softmax scores for unfamiliar inputs
• ReAct's fix: Clips excessively high activations before the softmax layer, preventing the inflated confidence that fools MSP

Uses the Helmholtz free energy function as a discriminative score, assigning low energy to in-distribution and high energy to OOD samples. Complements ReAct's activation-level intervention.

• Energy formula: E(x) = -T · log(Σ exp(f_i(x)/T))
• Advantage over MSP: Energy scores are less saturated and more aligned with input density
• Synergy with ReAct: Both methods manipulate the logit space—ReAct clips activations while energy-based models rescale through temperature T

A two-stage OOD method combining temperature scaling and input perturbations to widen the gap between in-distribution and OOD softmax scores. Shares ReAct's philosophy of post-hoc score manipulation.

• Step 1: Applies temperature scaling (T > 1) to soften the softmax distribution
• Step 2: Adds small gradient-based perturbations to inputs before inference
• Contrast with ReAct: ODIN modifies inputs and temperature; ReAct modifies internal activations directly without input perturbation

A parametric detection method computing the distance of feature representations to class-conditional Gaussian distributions. Captures covariance structure that ReAct's activation clipping preserves.

• Computation: D_M(x) = min_c (f(x) - μ_c)ᵀ Σ⁻¹ (f(x) - μ_c)
• Generative assumption: Features follow multivariate normal distributions per class
• Relationship: ReAct operates on the penultimate layer activations that Mahalanobis methods analyze, making them complementary diagnostic tools

Leverages the observation that the gradient magnitude of KL divergence with respect to model parameters is higher for in-distribution data. Exploits the same activation patterns that ReAct rectifies.

• Key insight: OOD inputs produce smaller parameter gradients than in-distribution inputs
• Computation: Measures ||∂KL/∂W|| for a uniform target distribution
• Connection: ReAct's clipping of high activations affects the very gradient norms that GradNorm uses as its detection signal

A training-time strategy that leverages auxiliary outlier datasets to teach models heuristics for detecting unknown inputs. Complements ReAct's post-hoc approach with proactive training interventions.

• Training objective: Combines standard classification loss with an OOD detection auxiliary loss
• Outlier sources: Uses diverse datasets like 80 Million Tiny Images or synthetic noise
• Combined approach: Models trained with outlier exposure can apply ReAct at inference for a defense-in-depth OOD detection pipeline