Novelty Detection

A semi-supervised algorithm that learns a decision boundary around the training data, which is assumed to be all 'normal'. It maps data into a high-dimensional space and finds a hyperplane that separates the data from the origin with maximum margin. New data points falling outside this learned region are flagged as novel.

• Key Parameter: nu, which sets an upper bound on the fraction of training errors and a lower bound on the fraction of support vectors.
• Use Case: Detecting novel machine states in industrial equipment using only data from normal operation.

An unsupervised, tree-based algorithm that explicitly isolates anomalies instead of profiling normal data. It works on the principle that anomalies are 'few and different', making them easier to isolate. It builds an ensemble of isolation trees, where the path length from the root to a leaf node is shorter for anomalies.

• Mechanism: Randomly selects a feature and a split value to partition the data. The average path length across all trees forms the anomaly score.
• Advantage: Highly scalable with linear time complexity and low memory usage, making it suitable for high-dimensional datasets.

A deep learning, unsupervised technique using a neural network trained to reconstruct its input. The model is trained only on normal data, learning a compressed representation (the latent space). Novel instances, which the model has not learned to reconstruct well, result in a high reconstruction error.

• Architecture: The network is typically bottlenecked, forcing it to learn the most salient features of the 'normal' distribution.
• Thresholding: A statistical threshold (e.g., using the mean + 3 standard deviations of training reconstruction errors) is set to flag novelties.

A density-based algorithm that measures the local deviation of a data point's density compared to its neighbors. It calculates the reachability distance to define local density. A point with a substantially lower density than its neighbors is considered a novelty.

• Core Metric: The LOF score. A score approximately equal to 1 indicates density similar to neighbors. A score significantly greater than 1 indicates a novelty.
• Strength: Effectively identifies novelties in data where the density of normal instances is not uniform, handling clusters of varying densities.

A probabilistic model that assumes all normal data is generated from a mixture of a finite number of Gaussian distributions. The model is trained to learn the parameters (mean, covariance, weight) of these components. Novelty is detected by evaluating the log-likelihood of a new data point under the fitted model; points with very low likelihood are flagged.

• Flexibility: Can model complex, multi-modal normal distributions better than a single Gaussian.
• Use Case: Modeling normal network traffic patterns, where traffic can belong to several distinct behavioral clusters.

A simple, distance-based method where the novelty score for a point is defined as its distance to its k-th nearest neighbor in the training set. Alternatively, the average distance to all k-nearest neighbors can be used. The underlying assumption is that normal data points exist in dense neighborhoods, while novelties are far from their nearest neighbors.

• Critical Choice: The distance metric (Euclidean, Mahalanobis) and the value of k heavily influence performance.
• Consideration: Computationally expensive for large datasets, as it requires storing the entire training set and calculating pairwise distances for scoring.

Anomaly detection is the general process of identifying rare items, events, or observations that deviate significantly from the majority of data. Unlike novelty detection, it often assumes anomalies are present (though unlabeled) in the training data. Key approaches include:

• Statistical methods like Z-scores and the Interquartile Range (IQR).
• Machine learning algorithms such as Isolation Forest and Local Outlier Factor (LOF).
• Deep learning techniques like autoencoders trained to reconstruct normal data, where high reconstruction error indicates an anomaly.

One-Class Support Vector Machine (SVM) is a classic semi-supervised algorithm for novelty detection. It learns a decision function by fitting a tight boundary (a hypersphere or hyperplane) around the training data, which is assumed to be all "normal." New data points that fall outside this learned boundary are classified as novel or anomalous. It is particularly useful when only examples of the normal class are available for training, making it a direct methodological precursor to many modern novelty detection systems.

Outlier detection is often used synonymously with anomaly detection but with a stronger statistical connotation. It focuses on identifying data points that are numerically distant from the rest of the observations. These deviations can be due to:

• Measurement error or data corruption.
• Inherent variability in a process.
• A novel underlying process not represented in the main distribution. Methods are typically rule-based (e.g., IQR, Mahalanobis Distance) and detect point anomalies without necessarily modeling the "normal" state as explicitly as novelty detection does.

Concept drift is a fundamental challenge that novelty detection systems often monitor. It refers to the change in the statistical properties of the target variable a model is trying to predict, over time. This is different from detecting novel individual instances; concept drift signifies that the underlying relationship between inputs and outputs has evolved. Detecting it is crucial because it causes model performance degradation. Techniques include monitoring prediction error rates or statistical tests on feature distributions.

This distinction is critical for understanding novelty detection's place in the ML landscape:

• Unsupervised Anomaly Detection: Algorithms like DBSCAN or Isolation Forest analyze the entire dataset's structure without any labels, identifying points in low-density regions as anomalies. Anomalies are assumed to be present in the training data.
• Semi-Supervised Novelty Detection: Techniques like One-Class SVM or deep autoencoders are trained only on normal data. They learn a model of "normality" and subsequently flag any instance that deviates from this model as novel. This is the standard paradigm for pure novelty detection.

Autoencoders are a powerful deep learning approach used for both anomaly and novelty detection. The neural network is trained to compress input data (encode) and then reconstruct it (decode) with minimal error. When trained exclusively on normal data, it becomes proficient at reconstructing similar patterns. Anomalies or novel instances result in a high reconstruction error, as their patterns were not learned during training. The reconstruction error score is then used as an anomaly threshold. This method is highly effective for complex, high-dimensional data like images or sensor readings.