Machine Learning Anomaly Detection

An ensemble-based, unsupervised algorithm that isolates anomalies instead of profiling normal data. It works by randomly selecting a feature and a split value to recursively partition the data. Anomalies are data points that require fewer random partitions to be isolated.

• Key Mechanism: Constructs binary trees; the average path length from the root to a leaf node is the anomaly score.
• Advantages: Highly efficient on large datasets with sub-sampling, linear time complexity, and low memory footprint.
• Typical Use: Detecting spikes in system metrics, fraudulent transactions, or manufacturing defects in high-dimensional data.

A kernel-based, unsupervised model that learns a tight boundary around normal training data in a high-dimensional feature space. The algorithm's objective is to maximize the margin between the origin and the data, treating the origin as a synthetic outlier.

• Key Mechanism: Uses a kernel function (e.g., RBF) to map data; anomalies fall outside the learned decision boundary.
• Advantages: Effective for non-linear data distributions and robust to noise in the training set.
• Typical Use: Validating sensor data from a known-good machine, detecting novel network intrusion patterns, or verifying product quality where only normal examples are available for training.

A neural network-based, unsupervised model trained to reconstruct its input. It consists of an encoder that compresses data into a lower-dimensional latent representation and a decoder that reconstructs the original input. Anomalies are data points with high reconstruction error.

• Key Mechanism: The bottleneck layer forces the network to learn the most salient features of normal data; anomalies deviate from this learned distribution.
• Advantages: Can capture complex, non-linear patterns and temporal dependencies (with LSTM/GRU layers).
• Typical Use: Detecting anomalies in sequential data like server logs, video frames, or multivariate time-series from industrial IoT sensors.

A density-based, unsupervised algorithm that measures the local deviation of a data point's density compared to its neighbors. It identifies anomalies as points in regions of significantly lower density.

• Key Mechanism: Calculates a score based on the ratio of the local reachability density of a point and the densities of its k-nearest neighbors.
• Advantages: Can detect anomalies where the global distribution is not uniform and where anomalies form small clusters.
• Typical Use: Identifying localized data quality issues, such as a subset of malfunctioning sensors in a network, or detecting insider threats in user behavior analytics.

A parametric, unsupervised method that assumes normal data is generated from a multivariate Gaussian distribution. It fits a robust estimate of the data's covariance matrix to define an elliptical boundary.

• Key Mechanism: Uses the Minimum Covariance Determinant estimator to fit an ellipse that encompasses the central mass of the data; points falling outside are anomalies.
• Advantages: Provides a clear probabilistic foundation and is computationally efficient for parametric data.
• Typical Use: Quality control in manufacturing with normally distributed metrics, detecting outliers in financial portfolio returns, or cleaning training datasets for other models.

A density-based clustering algorithm often repurposed for anomaly detection. It groups together points that are closely packed (core points) and marks points in low-density regions as outliers.

• Key Mechanism: Defines clusters based on two parameters: epsilon (maximum distance between points) and min_samples. Points not assigned to any cluster are labeled noise (anomalies).
• Advantages: Does not require specifying the number of clusters and can find arbitrarily shaped clusters.
• Typical Use: Identifying noise in spatial data, detecting fraudulent credit card applications that don't conform to any known pattern, or cleaning geospatial datasets.

Statistical Anomaly Detection identifies outliers by modeling the expected statistical distribution of data and flagging points that fall in low-probability regions. Unlike ML models that learn complex patterns, this method relies on established parametric and non-parametric tests.

• Key Techniques: Z-score/standard deviation, Interquartile Range (IQR), Grubbs' test, and control charts (e.g., Shewhart charts).
• Use Case: Ideal for univariate data with a known distribution (e.g., Gaussian) or for establishing simple, interpretable thresholds on metrics like latency or error rates.
• Limitation: Assumes data is independently and identically distributed (i.i.d.) and struggles with high-dimensional, non-stationary data where ML methods excel.

Dynamic Baseline Calculation is the automated, continuous computation of expected normal ranges for metrics, which adapts to trends, seasonality, and data drift to improve anomaly detection accuracy. It provides the reference 'normal' state against which current data is compared.

• Core Function: Replaces static thresholds with adaptive bounds using methods like rolling windows, exponential smoothing (Holt-Winters), or seasonal decomposition.
• Integration: Serves as a critical pre-processing step for both statistical and ML-based detectors, ensuring alerts account for legitimate periodic changes (e.g., daily user activity cycles).
• Benefit: Dramatically reduces false positives caused by expected behavioral shifts, making anomaly detection systems more trustworthy.

An Isolation Forest is an unsupervised machine learning algorithm specifically designed for anomaly detection. It isolates observations by randomly selecting a feature and a split value, with the core assumption that anomalies are few, different, and require fewer random partitions to be isolated.

• Mechanism: Builds an ensemble of binary decision trees. The average path length from the root to a data point is used as an anomaly score; shorter paths indicate higher likelihood of being an anomaly.
• Advantages: Highly efficient on large, high-dimensional datasets, has low linear time complexity, and requires minimal memory as it does not rely on distance or density measures.
• Typical Application: Detecting fraudulent transactions, network intrusion attempts, and defective parts in manufacturing.

An Autoencoder is a neural network trained to reconstruct its input, used for anomaly detection by learning a compressed representation (encoding) of normal data. Anomalies are identified by their high reconstruction error.

• Architecture: Consists of an encoder that compresses the input into a latent-space representation and a decoder that reconstructs the input from this representation.
• Training: The model is trained exclusively on normal data. It learns to efficiently reconstruct normal patterns but fails to accurately reconstruct unseen, anomalous patterns.
• Use Case: Effective for complex, high-dimensional data like images, sensor telemetry, or system metrics where anomalies are subtle and non-linear.

A One-Class SVM is an unsupervised model that learns a tight boundary around normal training data in a high-dimensional feature space. Data points falling outside this boundary are classified as anomalies.

• Core Principle: Maps data into a higher-dimensional space using a kernel function (e.g., Radial Basis Function) and finds a maximal margin hyperplane that separates the data from the origin.
• Strengths: Effective in high-dimensional spaces and can capture complex boundaries due to the kernel trick. Robust to outliers in the training set.
• Practical Consideration: Can be computationally intensive for very large datasets and requires careful kernel and parameter selection.

Local Outlier Factor (LOF) is a density-based algorithm that calculates the local deviation of a given data point with respect to its neighbors. It identifies anomalies that are in low-density regions relative to their local neighborhood.

• How it works: Compares the local density of a point to the densities of its k-nearest neighbors. A point with a significantly lower density than its neighbors has a high LOF score and is considered an anomaly.
• Advantage over Global Methods: Can detect local anomalies that would appear normal in a global statistical context (e.g., a dense cluster within a sparse region).
• Common Application: Detecting anomalies in spatial data, identifying unusual patterns within specific user segments, or finding defective sensors in a network.