Weibull Calibration

Weibull Calibration is grounded in Extreme Value Theory (EVT), which models the statistical behavior of tail-end events. Rather than assuming a Gaussian distribution of distances, it fits a Weibull distribution to the largest distances between correctly classified training samples and their class means. This provides a principled, non-parametric way to model the boundary of class inclusion, directly quantifying the probability that a query sample belongs to a known class based on its distance in the feature embedding space.

The calibration process operates on the distance metric space of a trained neural network. For each known class, the algorithm:

• Computes the mean activation vector (class prototype) from training samples
• Calculates the Euclidean or Mahalanobis distance from each sample to its class mean
• Fits a Weibull distribution to the tail of these distances (largest 10-20%) This tail distribution models the probability that a sample at a given distance is a genuine member of the class, rejecting those that fall into the open space beyond.

Weibull Calibration is the core statistical engine behind the OpenMax algorithm, which replaces the standard SoftMax layer for open set recognition. OpenMax uses the per-class Weibull CDF to recalibrate activation vectors:

• It computes the probability of inclusion for each top-k class
• Unknown classes are assigned a pseudo-activation based on the cumulative rejection probability
• The final output is a probability distribution over K+1 classes, where the +1 represents the unknown This transforms a closed-set classifier into an open-set recognizer without retraining the feature extractor.

A critical hyperparameter in Weibull Calibration is the tail size—the percentage of largest distances used to fit the distribution. This parameter controls the trade-off between:

• Statistical stability: Larger tails include more data but may violate EVT assumptions
• Extreme value fidelity: Smaller tails better model true extremes but increase variance Typical values range from 10% to 25% of the training samples per class. Cross-validation on a held-out validation set with known and unknown classes is used to optimize this parameter for the target openness measure.

The canonical implementation of Weibull Calibration is found in the LibMR (Library for Meta-Recognition) framework. LibMR provides efficient routines for:

• Fitting Weibull distributions using maximum likelihood estimation
• Computing cumulative distribution function values for query distances
• Performing statistical tests for goodness-of-fit This library is the computational backbone of the original OpenMax implementation and has been integrated into numerous open set recognition pipelines for radio frequency fingerprinting and computer vision applications.

The primary purpose of Weibull Calibration is to manage open space risk—the risk of labeling an unknown emitter as a known class. By modeling the tail of the distance distribution, the technique quantifies how far a sample can be from a class mean before it should be rejected. This provides a statistically rigorous threshold that adapts to each class's natural variance:

• Compact classes have tight Weibull fits with sharp rejection boundaries
• Dispersed classes have broader fits reflecting higher intra-class variance This per-class calibration is essential for dynamic spectrum environments where emitter signatures vary in consistency.

The statistical discipline underpinning Weibull Calibration. EVT models the probability of rare, extreme events—specifically, the tail behavior of distance distributions. The Weibull distribution is one of three extreme value distributions, making it ideal for modeling the probability of inclusion for samples far from class means. In open set recognition, EVT provides the theoretical justification for rejecting inputs that fall into the distributional tail.

A critical distance metric often used as input to Weibull Calibration. Unlike Euclidean distance, Mahalanobis distance accounts for feature covariance, measuring how many standard deviations a sample is from the class mean along each principal axis. This produces a more statistically meaningful distance for EVT modeling, as it respects the correlation structure of the learned feature space rather than assuming spherical class distributions.

The precise risk that Weibull Calibration is designed to mitigate. Defined as the volume of feature space far from any training data that is nonetheless labeled as a known class. Weibull Calibration quantifies this risk by modeling the probability that a sample at a given distance belongs to a class, enabling a principled threshold for rejection. The goal is to bound open space risk while maintaining high accuracy on known classes.

A broader framework that Weibull Calibration extends to open set scenarios. Standard calibration aligns predicted probabilities with empirical accuracy using methods like Temperature Scaling. Weibull Calibration adds a spatial dimension: it calibrates confidence based on distance from training data, not just SoftMax magnitude. This spatial awareness is essential for distinguishing between high-confidence correct predictions and high-confidence errors on unknown inputs.

The low-dimensional vector space where Weibull Calibration operates. Neural networks learn to map high-dimensional RF signals into compact embeddings where semantic similarity equals geometric proximity. Weibull Calibration then models the distribution of distances within this space:

• Class means serve as prototypical anchors
• Distance distributions characterize class boundaries
• Tail behavior determines rejection thresholds