Mahalanobis Distance

Unlike Euclidean distance, which treats all dimensions equally, the Mahalanobis distance scales the separation along each principal component by the inverse of the covariance matrix. This effectively transforms the feature space into a unit-variance sphere, ensuring that a deviation along a low-variance, highly correlated direction is weighted more heavily than the same deviation along a high-variance, noisy axis. This prevents the classifier from being misled by spurious correlations in the IQ sample or cumulant feature space.

In open set recognition, the Mahalanobis distance serves as a powerful confidence score. A deep network is first trained on known modulation classes. The class-conditional Gaussian distributions are then fitted to the learned embeddings by computing a class mean vector and a shared or per-class covariance matrix. At inference, the minimum Mahalanobis distance from a query sample to any known class centroid is calculated. If this distance exceeds a calibrated threshold, the sample is flagged as an unknown or novel modulation.

Using the Mahalanobis distance for classification is mathematically equivalent to a Gaussian Discriminant Analysis classifier with a shared covariance matrix. The decision boundary becomes a linear hyperplane. This property bridges discriminative deep learning with generative probability density estimation, allowing the model to not only predict a class label but also estimate the likelihood of the input belonging to the training distribution, which is critical for detecting distributional shift.

The standard empirical covariance estimator is highly sensitive to outliers and can be singular in high-dimensional feature spaces with limited samples. For reliable Mahalanobis distance calculation, robust estimators are essential:

• Minimum Covariance Determinant: Finds an outlier-free subset of the data to compute a stable covariance matrix.
• Shrinkage Estimators: Combine the sample covariance with a structured target matrix to reduce estimation error and guarantee invertibility.
• Empirical Bayes Ridge: Adds a regularization term to the diagonal of the covariance matrix, stabilizing the inverse.

A key preprocessing step for effective Mahalanobis-based detection is deep whitening. After the final embedding layer, a transformation matrix derived from the inverse square root of the covariance matrix is applied. This linear operation decorrelates the features and normalizes their variances to unity. In this whitened space, the Mahalanobis distance reduces to a simple Euclidean distance, allowing for computationally efficient nearest-centroid classification and novelty detection without repeated matrix inversions.

The Mahalanobis distance relies on the assumption that the class-conditional data is unimodal and Gaussian. In the highly nonlinear embedding spaces of deep networks, this assumption often breaks down. The distance can also suffer from the curse of dimensionality, where the volume of the covariance ellipsoid concentrates at a thin shell, making all points appear equidistant. For complex, multi-modal signal distributions, a mixture of Gaussians or a non-parametric k-Nearest Neighbor distance in the whitened space often provides a more reliable novelty score.

The core statistical object that differentiates Mahalanobis distance from Euclidean distance. It captures the variance of each feature and the correlation between feature pairs in a dataset.

• Defines the shape and orientation of a class's distribution.
• The Mahalanobis distance effectively uses the inverse of this matrix to whiten or decorrelate the data.
• In signal classification, it models the natural spread of IQ samples for a specific modulation scheme under noise.

The straight-line, 'as-the-crow-flies' distance between two points. It assumes all features are equally important and completely uncorrelated.

• Computationally simple but statistically naive for correlated data.
• Fails to recognize that a point far from the mean in a low-variance direction is more anomalous than a point equally far in a high-variance direction.
• Mahalanobis distance reduces to Euclidean distance when the covariance matrix is the identity matrix.

The primary application of Mahalanobis distance in this domain. A sample is flagged as unknown if its Mahalanobis distance to the nearest known class prototype exceeds a calibrated threshold.

• Provides a principled statistical test for novelty.
• More robust than SoftMax confidence scores, which can be arbitrarily high for unknown inputs.
• Often used to fit a class-conditional Gaussian distribution in a learned feature space.

A preprocessing transformation that decorrelates data and scales it to unit variance. Computing the Euclidean distance on whitened data is mathematically equivalent to computing the Mahalanobis distance on the original data.

• Involves multiplying data by the inverse square root of the covariance matrix.
• Creates an isotropic, spherical distribution where standard distance metrics become valid.
• A key step in many deep open-set classifiers to simplify the geometry of the embedding space.

The theoretical probability distribution that squared Mahalanobis distances follow if the original data is multivariate Gaussian. This allows for a rigorous statistical threshold to be set for novelty detection.

• The degrees of freedom equal the dimensionality of the feature space.
• A threshold can be set based on a desired confidence level (e.g., 99.9%) to reject outliers.
• Deviation from this distribution in the tail can be modeled using Extreme Value Theory for more accurate open-set calibration.

A classification strategy where the network learns a single representative embedding vector, or prototype, for each known modulation class. The Mahalanobis distance to these prototypes serves as the direct decision metric.

• Replaces a linear classification layer with a distance-based one.
• Enforces a compact, Gaussian-like feature distribution around each prototype during training.
• Enables open-set rejection by simply thresholding the minimum distance to any known prototype.