Mahalanobis Distance

Unlike Euclidean distance, which treats all dimensions equally, the Mahalanobis distance transforms the space using the inverse of the covariance matrix. This effectively rescales axes by their standard deviations and rotates them to remove correlation. A distance of 1.0 always means a point is exactly one standard deviation away from the mean along the principal component of the distribution, regardless of the original feature scales.

The squared Mahalanobis distance of a vector x from a distribution with mean μ and covariance matrix Σ is defined as: D² = (x - μ)ᵀ Σ⁻¹ (x - μ).

• If Σ is the identity matrix, the formula collapses to the standard Euclidean distance.
• If Σ is diagonal, it becomes the normalized Euclidean distance, scaling each feature by its variance.
• The full matrix Σ⁻¹ accounts for off-diagonal covariances, penalizing deviations that go against the natural correlation of the data.

In high-dimensional spaces, Euclidean distances tend to concentrate, making outliers hard to detect. The Mahalanobis distance provides a statistically principled threshold. By assuming the data is approximately multivariate Gaussian, the squared distances follow a chi-squared (χ²) distribution. A point with a distance exceeding a critical χ² value (e.g., for p=0.975) can be flagged as a multivariate outlier, a technique widely used in multivariate process control and fraud detection.

The classic Mahalanobis distance suffers from the masking effect, where multiple outliers distort the sample mean and covariance matrix, making them appear normal. To counter this, Minimum Covariance Determinant (MCD) estimation finds a subset of h observations whose covariance matrix has the smallest determinant. The distance is then computed using this robust location and scatter matrix, allowing for reliable outlier detection even when up to 50% of the data is contaminated.

Applying the transformation z = Σ⁻¹/² (x - μ) is known as whitening or sphering. The resulting vector z has a zero mean and an identity covariance matrix. In this transformed space, the Euclidean distance of z from the origin is exactly the Mahalanobis distance of x from μ. This is a critical preprocessing step for many machine learning algorithms, including Independent Component Analysis (ICA) and certain neural network architectures, as it removes first and second-order statistical dependencies.

The calculation requires inverting the covariance matrix Σ. When the number of features (p) exceeds the number of samples (n), or when features are perfectly collinear, Σ becomes singular and non-invertible. In such cases, the pseudo-inverse (Moore-Penrose) is used, or dimensionality reduction via Principal Component Analysis (PCA) is applied first. The distance is then computed on the retained principal components, effectively regularizing the estimate in the n < p scenario.

The mathematical engine that powers the Mahalanobis distance. This square matrix captures the pairwise covariances between all features in a dataset.

• Diagonal elements represent the variance of each individual feature.
• Off-diagonal elements represent how two features vary together. By inverting this matrix, the Mahalanobis distance effectively whitens the data, transforming correlated axes into a standardized, isotropic space where Euclidean distance becomes meaningful.

The primary application of the Mahalanobis distance in high-stakes environments. Unlike univariate methods that flag points based on a single dimension, this technique identifies anomalies that are only visible in a multivariate context.

• A data point can be within a normal range for every individual feature yet still be a highly anomalous combination.
• The distance squared follows a chi-squared distribution, allowing engineers to set a statistically rigorous threshold for flagging outliers.

The multivariate generalization of the Student's t-test, directly derived from the squared Mahalanobis distance. It is used to test whether the mean vector of a sample differs significantly from a hypothesized population mean.

• Essential in statistical process control for monitoring multiple correlated quality metrics simultaneously.
• Detects a shift in the process mean vector before any single variable triggers a univariate control chart alarm.

The simpler, isotropic counterpart to the Mahalanobis distance. It measures the straight-line distance between two points but assumes all features are uncorrelated and have equal variance.

• Using Euclidean distance on correlated data creates misleading neighborhoods where a point appears closer to a distant cluster than to its own.
• The Mahalanobis distance reduces to the Euclidean distance if and only if the covariance matrix is the identity matrix.

The 'reducible' uncertainty arising from a lack of knowledge about the true data-generating process. The Mahalanobis distance is a powerful tool for quantifying this in the input space.

• A high Mahalanobis distance to the training manifold signals that a test point is in a sparse region of the feature space.
• This metric is often used in active learning to query labels for the most novel, out-of-distribution samples where the model's epistemic uncertainty is highest.

A preprocessing step that decorrelates and standardizes features, turning the Mahalanobis distance into a simple Euclidean distance in the transformed space.

• The transformation matrix is the inverse square root of the covariance matrix.
• After applying this linear transformation, the data has an identity covariance matrix, making it a critical normalization technique for algorithms that assume spherical input distributions.