Outlier Detection

These methods assume the underlying data follows a known probability distribution (e.g., Gaussian). Outliers are identified as points with a low probability of belonging to the fitted model.

• Z-Score/Standard Deviation: Flags points beyond a threshold (e.g., ±3 standard deviations) from the mean.
• Interquartile Range (IQR): A robust non-parametric method where outliers are points below Q1 - 1.5IQR or above Q3 + 1.5IQR.
• Grubbs' Test: A statistical test for detecting a single outlier in a univariate dataset assumed to be normally distributed.

Best for univariate data or when the distribution is well-understood.

These techniques identify outliers based on the density or distance of data points relative to their neighbors, without assuming a specific distribution.

• k-Nearest Neighbors (k-NN): Scores a point by the distance to its k-th nearest neighbor; isolated points have higher scores.
• Local Outlier Factor (LOF): Measures the local density deviation of a point compared to its neighbors. Points in low-density regions are outliers.
• DBSCAN (Density-Based Clustering): A clustering algorithm that inherently labels points in low-density regions as noise (a form of outlier).

Effective for multivariate data where clusters have varying densities.

These methods project data into lower-dimensional spaces and identify points that deviate from the projected structure.

• Principal Component Analysis (PCA): Outliers exhibit large reconstruction error when projected back from the principal component subspace.
• One-Class Support Vector Machines (SVM): Learns a tight boundary around 'normal' data in a high-dimensional feature space; points outside are anomalies.

Useful for high-dimensional data where the normal data lies near a linear subspace or a defined region.

These algorithms isolate outliers by randomly partitioning data. Outliers are points that are easier to isolate.

• Isolation Forest: Builds an ensemble of random decision trees. The average path length to isolate a point is the anomaly score; outliers have shorter paths.
• Random Cut Forest (RCF): A streaming variant of Isolation Forest designed for real-time anomaly detection on data streams.

Highly efficient for large, high-dimensional datasets and does not require a distance metric.

Specialized for temporal data, these techniques model sequences and flag points or subsequences that break temporal patterns.

• Moving Average & Standard Deviation: Flags points deviating from a local trend.
• Seasonal-Trend Decomposition (STL): Identifies outliers in the residual component after removing trend and seasonality.
• Deep Learning (LSTM Autoencoders): Neural networks learn to reconstruct normal sequences; high reconstruction error indicates an anomaly.

Crucial for monitoring metrics, financial transactions, sensor telemetry, and log data.

These advanced concepts refine what constitutes an anomaly.

• Contextual Outliers: A point is anomalous within a specific context (e.g., time of day, location). Requires defining contextual and behavioral attributes.
• Collective Outliers: A collection of related data points is anomalous, even if individual points are normal (e.g., a sudden burst of failed login attempts).

These require more sophisticated modeling but are critical for detecting fraud, cyber-attacks, and system faults.

Anomaly detection is the broader category of techniques for identifying rare items, events, or observations that raise suspicions by differing significantly from the majority of the data. While often used interchangeably with outlier detection, anomaly detection frequently implies a focus on temporal sequences or system behavior.

• Key Distinction: Outlier detection is often applied to static datasets, while anomaly detection is commonly used for real-time, time-series data streams (e.g., server metrics, transaction volumes).
• Core Methods: Include statistical tests, density-based methods, and machine learning models like Isolation Forests or Autoencoders.
• Primary Goal: To flag data points that do not conform to an expected pattern, indicating potential issues like fraud, network intrusion, or mechanical failure.

Data drift detection is the process of identifying and measuring changes in the statistical properties of production data over time compared to a baseline or training data. It is a critical form of monitoring for machine learning systems.

• Focus: Detects shifts in the input data distribution (covariate shift), the relationship between input and output (concept drift), or the target variable distribution (prior probability shift).
• Impact: Unmitigated data drift degrades model performance silently, as the model's assumptions about the data become invalid.
• Common Techniques: Uses statistical distance measures (e.g., Population Stability Index, Kullback–Leibler divergence) and hypothesis tests (e.g., Kolmogorov-Smirnov) to quantify drift.

Statistical Process Control (SPC) is a method of quality control which employs statistical methods to monitor and control a process. In data observability, SPC principles are applied to data generation pipelines.

• Core Tool: The control chart, which plots process data over time with statistically derived upper and lower control limits.
• Application: Used to distinguish between common-cause variation (inherent to the process) and special-cause variation (indicative of an anomaly or outlier in the data generation mechanism).
• Key Metrics: Monitors metrics like row counts, null percentages, or aggregate sums, treating them as the "output" of a data manufacturing process.

Isolation Forest is an unsupervised machine learning algorithm specifically designed for efficient outlier detection. It works on the principle of isolating anomalies rather than profiling normal data points.

• Mechanism: It recursively partitions data by randomly selecting a feature and a split value. Outliers are 'isolated' closer to the root of the tree, requiring fewer random partitions.
• Efficiency: Has a linear time complexity and low memory requirement, making it suitable for high-dimensional datasets.
• Output: Assigns an anomaly score to each data point. A score close to 1 indicates an anomaly, a score much smaller than 0.5 indicates a normal point.

The Z-score (or standard score) is a fundamental statistical measure that quantifies how many standard deviations a raw data point is from the population mean. It is a basic, parametric method for univariate outlier detection.

• Calculation: (Z = \frac{(X - \mu)}{\sigma}), where (X) is the raw score, (\mu) is the population mean, and (\sigma) is the population standard deviation.
• Thresholding: Data points with an absolute Z-score greater than a threshold (commonly 2, 2.5, or 3) are flagged as potential outliers.
• Assumption: Relies on the assumption that the data is normally distributed. It is less effective for skewed distributions or multivariate data without modification.

Local Outlier Factor (LOF) is a density-based unsupervised algorithm for outlier detection that measures the local deviation of a given data point's density compared to its neighbors.

• Core Concept: Instead of viewing outliers as global phenomena, LOF identifies points that are outliers relative to their local neighborhood. A point in a sparse region may be a normal point if its local neighborhood is also sparse.
• Process: Calculates the reachability distance and local reachability density for each point, then computes the LOF score as a ratio of the average local density of its k-nearest neighbors to its own local density.
• Output: An LOF score approximately equal to 1 indicates density similar to neighbors. A score significantly greater than 1 indicates an outlier.