Temperature Scaling

Temperature scaling introduces a single scalar parameter, T > 0, that divides the logits before the SoftMax function. This parameter is learned by minimizing the Negative Log Likelihood (NLL) on a held-out validation set. A T > 1 softens the probability distribution, reducing overconfidence, while T < 1 sharpens it. Crucially, because T is a monotonic transformation of the logits, the maximum predicted class and the model's accuracy remain completely unchanged.

Temperature scaling is the simplest member of a family of matrix scaling methods. More complex variants learn a full linear transformation of the logits:

• Vector Scaling: Applies a separate temperature parameter for each class.
• Matrix Scaling: Learns a full weight matrix and bias vector to transform logits. While more expressive, these methods require significantly more calibration data and can overfit, often failing to outperform the single-parameter temperature scaling on standard benchmarks like CIFAR-100 and ImageNet.

The primary metric for evaluating temperature scaling is the Expected Calibration Error (ECE). ECE measures the discrepancy between a model's predicted confidence and its empirical accuracy. It is computed by partitioning predictions into M bins based on confidence, then taking a weighted average of the absolute difference between accuracy and confidence within each bin. A perfectly calibrated model has an ECE of zero. Temperature scaling is explicitly optimized to minimize this metric.

A defining characteristic of temperature scaling is its accuracy-preserving property. Because the operation argmax(softmax(logits / T)) is invariant to the scalar T (for T > 0), the model's top-1 classification decision never changes. This makes it a strictly post-hoc calibration tool that can be applied to any existing trained model without any risk of degrading its discriminative performance, a critical advantage for safety-critical systems where retraining is impossible.

The optimal temperature T is found by minimizing the Negative Log Likelihood (NLL) on a calibration set, which is a proper scoring rule. This is typically done using gradient descent or a simple line search. The process is computationally trivial compared to model training. The learned T directly addresses the root cause of miscalibration in modern networks: the tendency of cross-entropy loss to push logits to extreme magnitudes, resulting in peaked, overconfident SoftMax outputs.

The process of aligning a model's predicted probability of correctness with its actual empirical accuracy. A perfectly calibrated model that predicts 70% confidence will be correct exactly 70% of the time. Temperature scaling is the most common post-hoc calibration method, but alternatives include Platt scaling (fitting a logistic regression on logits) and isotonic regression (a non-parametric approach). In open set emitter recognition, calibration is critical: poorly calibrated probabilities lead to incorrect rejection thresholds, causing unknown emitters to be classified as known with high confidence.

• Expected Calibration Error (ECE) quantifies the miscalibration gap
• Reliability diagrams visually plot confidence against observed accuracy
• Modern neural networks tend toward overconfidence on both in-distribution and out-of-distribution inputs

The reducible model uncertainty arising from a lack of knowledge or data. This type of uncertainty is high for inputs far from the training distribution—exactly the scenario encountered with unknown emitter classes. Unlike aleatoric uncertainty (irreducible noise in the data itself), epistemic uncertainty can be reduced by collecting more training samples. Temperature scaling primarily addresses confidence calibration but does not directly model epistemic uncertainty.

• Monte Carlo Dropout approximates epistemic uncertainty through multiple stochastic forward passes
• Deep Ensembles capture model uncertainty by averaging predictions across independently trained networks
• Evidential Deep Learning places a Dirichlet prior over class probabilities to jointly model belief mass and uncertainty

A framework that learns an energy function assigning low energy to in-distribution data and high energy to out-of-distribution data. Unlike temperature scaling, which only adjusts the SoftMax temperature parameter, EBMs provide a principled approach to open set rejection by establishing an energy landscape. Inputs with energy scores above a calibrated threshold are rejected as unknown.

• The Helmholtz free energy formulation connects SoftMax logits to an energy score: E(x) = -T · log Σ exp(logit_i / T)
• EBMs naturally separate the tasks of classification (finding the lowest energy class) and detection (thresholding the energy score)
• Joint training with contrastive divergence can shape the energy landscape to explicitly push unknown samples to high-energy regions

An algorithm that replaces the standard SoftMax layer with a mechanism calibrated using Extreme Value Theory (EVT) to estimate the probability of an unknown class. While temperature scaling adjusts the SoftMax temperature globally, OpenMax fits a Weibull distribution to the distance between each sample and its class mean in the penultimate layer's activation space. This enables explicit modeling of the probability that a sample belongs to no known class.

• Weibull calibration models the tail behavior of distances for each known class
• The algorithm recalibrates the SoftMax vector by redistributing probability mass to an explicit unknown class when activation patterns are far from all known class means
• OpenMax provides stronger theoretical guarantees for open set rejection than temperature scaling alone

A distribution-free framework that produces prediction sets with a guaranteed marginal coverage probability. Rather than outputting a single calibrated probability, conformal prediction generates a set of plausible class labels with a statistical guarantee that the true label is included with a user-specified confidence level (e.g., 95%). For open set emitter recognition, an empty prediction set signals an unknown class.

• Uses a calibration dataset to compute nonconformity scores that measure how unusual a new sample is relative to known classes
• Inductive conformal prediction avoids retraining by splitting the calibration set from the training data
• Provides rigorous finite-sample validity guarantees without assuming any distributional form, unlike temperature scaling which assumes the SoftMax parametric form

A Bayesian approximation technique that applies dropout at inference time to generate multiple stochastic forward passes, estimating model uncertainty for unknown inputs. By running N forward passes with dropout enabled and computing the predictive entropy or mutual information across the resulting probability distributions, the model can distinguish between epistemic uncertainty (high variance across passes) and aleatoric uncertainty (consistently uncertain predictions).

• Predictive entropy: H[1/N Σ p(y|x, ω_n)] measures total uncertainty
• Mutual information: H[predictive] - 1/N Σ H[p(y|x, ω_n)] isolates epistemic uncertainty
• Unlike temperature scaling, which only adjusts the SoftMax temperature, MC Dropout provides a richer uncertainty decomposition for open set rejection decisions