Time-Series Anomaly Detection

Classical approaches that model the normal distribution of data to flag deviations. These methods are computationally efficient and highly interpretable.

• Z-Score: Measures how many standard deviations a point is from the mean. A threshold of 3 is a common rule of thumb for flagging anomalies.
• ARIMA: Fits an autoregressive integrated moving average model to historical data and flags points that fall outside the prediction confidence interval.
• Seasonal Decomposition: Deconstructs a time series into trend, seasonal, and residual components using algorithms like STL. Anomalies are identified by analyzing the residual component after removing expected patterns.

Algorithms that learn complex, non-linear patterns from historical data without explicit programming. They excel at multivariate and contextual anomaly detection.

• Isolation Forest: An unsupervised algorithm that explicitly isolates anomalies instead of profiling normal points. It exploits the fact that anomalies are 'few and different,' requiring fewer random splits to isolate.
• One-Class SVM: Trained exclusively on 'normal' data to learn a decision boundary that encapsulates the majority of the dataset. Any point falling outside this boundary is classified as an anomaly.
• DBSCAN: A density-based clustering algorithm that groups closely packed points and identifies points in low-density regions as outliers, without requiring a pre-specified number of clusters.

Neural network architectures designed to model high-dimensional, sequential data. They automatically learn hierarchical feature representations, making them powerful for complex telemetry streams.

• Autoencoders: Trained to reconstruct normal data through a bottleneck layer. The reconstruction error—the difference between input and output—serves as an anomaly score. High error indicates a deviation from learned normality.
• Long Short-Term Memory (LSTM): A recurrent neural network capable of learning long-term dependencies. LSTMs can predict the next expected value in a sequence; significant deviations from the prediction are flagged as anomalies.

Static thresholds fail in dynamic environments where data distributions shift over time. Adaptive techniques automatically adjust anomaly boundaries based on recent statistical properties.

• Dynamic Thresholding: Uses rolling statistics (e.g., moving average and standard deviation) to compute thresholds that adapt to trends and seasonality.
• Concept Drift vs. Data Drift: It is critical to distinguish between a genuine anomaly and a legitimate shift in the data's underlying distribution. Concept drift changes the relationship between input and target, while data drift changes the input distribution itself, both requiring model adaptation.

Standard accuracy is misleading for highly imbalanced anomaly detection datasets where anomalies are rare. Specialized metrics are required.

• Precision-Recall AUC: The area under the precision-recall curve summarizes the trade-off between precision (how many flagged anomalies were real) and recall (how many real anomalies were found). It is more informative than ROC AUC for imbalanced data.
• Alert Fatigue: A critical operational metric. A model with high recall but low precision generates excessive false positives, desensitizing operations teams and risking missed critical incidents.

Not all anomalies are single, outlying points. Advanced detection must identify complex patterns.

• Contextual Anomaly: A value that is normal in one context but anomalous in another. For example, a server load of 80% is normal during peak hours but anomalous at 3:00 AM.
• Collective Anomaly: A sequence of data points that is anomalous as a whole, even if each individual point appears normal. A classic example is a sequence of small, repeated transactions that collectively form a fraud pattern.