Mahalanobis Distance

The Mahalanobis distance (D²) between a multivariate observation vector (x) and a distribution with mean vector (\mu) and covariance matrix (\Sigma) is defined as:

[ D^2 = (x - \mu)^T \Sigma^{-1} (x - \mu) ]

• (x - \mu): The deviation vector from the mean.
• (\Sigma^{-1}): The inverse of the covariance matrix, which scales and decorrelates the data.
• The result is a scalar distance that follows a chi-squared distribution under multivariate normality, allowing for statistical significance testing.

Unlike Euclidean distance, Mahalanobis distance incorporates the covariance structure of the data. This is its defining characteristic.

• Correlation Adjustment: If two variables are highly correlated, a point deviating in both is less surprising than if the variables were independent. The covariance matrix inverse down-weights distances along correlated directions.
• Variance Scaling: It automatically scales each variable by its variance, preventing features with larger numerical ranges from dominating the distance calculation. This makes it scale-invariant.
• Elliptical Contours: Points with equal Mahalanobis distance from the mean form ellipsoids (not spheres) aligned with the data's principal components, accurately reflecting the data's inherent geometry.

Mahalanobis distance is unitless and scale-invariant. This is a critical advantage for real-world datasets.

• Problem with Euclidean Distance: If one feature is measured in dollars (range 0-1,000,000) and another in percentages (0-100), Euclidean distance will be overwhelmingly influenced by the dollar feature.
• Mahalanobis Solution: By pre-multiplying by (\Sigma^{-1}), the distance internally standardizes all variables. A one-unit change in a high-variance feature contributes less to the final distance than a one-unit change in a low-variance feature.
• Practical Implication: Features can be in their native units without manual normalization, simplifying preprocessing for anomaly detection pipelines.

Under the assumption that the data follows a multivariate normal distribution, the squared Mahalanobis distance follows a chi-squared distribution.

• Distribution: ( D^2 \sim \chi^2_p ), where (p) is the number of variables (degrees of freedom).
• Threshold Selection: A statistically rigorous threshold for outlier detection is set using the chi-squared distribution's critical value. For example, with (p=5) variables and a 99% confidence level, the threshold is (\chi^2_{5, 0.99} \approx 15.09). Any point with (D^2 > 15.09) is flagged as an outlier.
• Limitation: This property is exact only for multivariate normal data. For non-normal data, the threshold may be estimated empirically (e.g., using the 99th percentile of calculated distances).

The accuracy of Mahalanobis distance is highly dependent on a reliable estimate of the covariance matrix (\Sigma).

• Classical Estimator: The sample covariance matrix is sensitive to outliers itself. A few extreme points can inflate variances and distort correlations, "masking" themselves and other outliers—a problem known as the masking effect.
• Robust Alternatives: For more reliable outlier detection, robust estimators of covariance are used:
  • Minimum Covariance Determinant (MCD): Finds a subset of the data that minimizes the determinant of the covariance matrix.
  • Minimum Volume Ellipsoid (MVE): Finds the smallest volume ellipsoid covering a subset of the data.
• Using these robust methods prevents outliers from influencing the very metric designed to find them.

Mahalanobis distance is a fundamental tool in multivariate statistical process control and machine learning.

• Industrial Quality Control: Monitoring multiple sensor readings (e.g., temperature, pressure, vibration) from a manufacturing process. A high D² indicates a faulty batch or machine state.
• Financial Fraud: Analyzing a transaction across features like amount, location, time, and merchant category. Unusual combinations, measured by D², can signal fraud.
• Health Informatics: Identifying abnormal patient profiles based on a suite of lab results (e.g., blood panel).
• Preprocessing for ML: Flagging and removing multivariate outliers from training data to improve model robustness and performance.
• Related Technique: It is the distance metric at the heart of Hotelling's T² statistic, the multivariate generalization of the t-test.

Euclidean distance is the straight-line distance between two points in Euclidean space. It is the most intuitive distance measure, calculated as the square root of the sum of squared differences between coordinates. In anomaly detection, it is effective for isotropic data (where variability is equal in all directions) but fails when features are correlated or have different scales, as it treats all dimensions equally. For example, measuring the distance between two data points (x1, y1) and (x2, y2) in a 2D plane uses the formula: √((x2 - x1)² + (y2 - y1)²).

• Key Limitation: It does not account for the correlation structure of the data.
• Contrast with Mahalanobis: Mahalanobis distance is a correlation-adjusted Euclidean distance, making it superior for multivariate, correlated datasets.

A Z-score measures how many standard deviations a single data point is from the mean of its distribution. It is defined as z = (x - μ) / σ, where x is the data point, μ is the mean, and σ is the standard deviation. It is the univariate equivalent of the Mahalanobis distance.

• Univariate Detection: Used to flag outliers in a single feature by identifying points where |z| > 3 (a common threshold).
• Foundation for Multivariate: The Mahalanobis distance generalizes this concept to multiple dimensions by incorporating the covariance matrix, which accounts for feature variances and their correlations. While a Z-score assumes a normal distribution for a single variable, Mahalanobis distance assumes a multivariate normal distribution.

The covariance matrix is a square matrix that encapsulates the variances of individual variables and the covariances between every pair of variables in a dataset. It is the core mathematical object that the Mahalanobis distance uses to normalize and decorrelate the data.

• Diagonal Elements: Represent the variance (spread) of each feature.
• Off-Diagonal Elements: Represent the covariance (directional relationship) between two features.
• Role in Mahalanobis Distance: The distance formula D² = (x - μ)ᵀ Σ⁻¹ (x - μ) uses the inverse of the covariance matrix (Σ⁻¹). This inversion effectively stretches and rotates the coordinate space so that the data becomes spherical (uncorrelated with unit variance), allowing for a meaningful distance calculation. If features are independent, the covariance matrix is diagonal, and Mahalanobis distance reduces to a scaled Euclidean distance.

The multivariate normal distribution is a generalization of the one-dimensional normal distribution to higher dimensions. It is defined by a mean vector (center) and a covariance matrix (shape and orientation). The Mahalanobis distance is intrinsically linked to this distribution.

• Distance as Probability: For data following a multivariate normal distribution, the squared Mahalanobis distance follows a chi-squared distribution. This allows for probabilistic interpretation and threshold setting (e.g., a point with D² > χ²(0.975) for p degrees of freedom can be considered an outlier with 97.5% confidence).

• Elliptical Contours: Points with an equal Mahalanobis distance from the mean lie on an ellipsoid, whose shape is determined by the covariance matrix. This makes it the natural distance measure for this family of distributions.

Robust Mahalanobis distance is a variation that uses robust estimates of the data's center (mean) and spread (covariance) to reduce the influence of outliers on the distance calculation itself. The standard Mahalanobis distance is sensitive to masking, where outliers distort the mean and covariance estimates, making them harder to detect.

• Robust Estimators: Employs the Minimum Covariance Determinant (MCD) or similar algorithms to estimate location and scatter from a clean subset of the data.
• Use Case: Essential for real-world datasets contaminated with outliers. It provides a more reliable distance measure by ensuring the underlying distribution parameters are not skewed by the anomalies it seeks to find. Libraries like scikit-learn offer the EllipticEnvelope which implements this robust method.

Isolation Forest is an unsupervised, tree-based algorithm specifically designed for efficient anomaly detection. Unlike distance-based methods, it isolates anomalies by randomly partitioning data. It operates on the principle that anomalies are few and different, making them easier to isolate with fewer random splits.

• Mechanism: It builds an ensemble of isolation trees. The path length (number of splits required to isolate a point) is the anomaly score; shorter paths indicate higher anomaly likelihood.
• Contrast with Mahalanobis: While Mahalanobis defines normality via a parametric distribution (multivariate normal), Isolation Forest is non-parametric and makes no assumptions about data distribution. It often excels with high-dimensional data and is less sensitive to correlated features by design, but may not provide the same probabilistic grounding as Mahalanobis distance.