Novelty Detection

The foundational assumption of novelty detection is that the training set contains zero anomalies. The model learns a precise boundary around only normal instances, making it highly sensitive to any deviation. This contrasts with outlier detection, where the training set may contain some contamination. In financial fraud, this means training on verified legitimate transactions from a known clean period to establish a baseline of normal customer behavior.

The model constructs a decision function that returns +1 for normal regions and -1 for novel regions. Key approaches include:

• One-Class SVM: Finds a hyperplane that maximally separates normal data from the origin in kernel space
• Deep SVDD: Maps normal data into a minimal hypersphere in latent space
• Gaussian Mixture Models: Fits a probability density to normal data and flags low-density regions
• Autoencoders: Learn to compress and reconstruct normal patterns, flagging high reconstruction error as novel

This distinction is critical for model selection:

• Novelty Detection: Training data is pristine (no anomalies). The model learns what is normal and flags anything unfamiliar. Used when you have a verified clean dataset of legitimate transactions.
• Outlier Detection: Training data may contain anomalies. The model must robustly identify outliers within potentially contaminated data. Used when historical data cannot be guaranteed clean.
• Key implication: Novelty detection models are more sensitive but less robust to training contamination than outlier detection methods.

Autoencoder-based novelty detection relies on reconstruction error as the anomaly score. The principle: a model trained only on normal data will reconstruct normal patterns well but fail on novel patterns. Key metrics include:

• Mean Squared Error (MSE) between input and reconstruction
• Reconstruction probability from Variational Autoencoders, which is more principled than raw error
• Mahalanobis distance in the latent space to measure deviation from the normal distribution In fraud detection, a transaction that cannot be well-reconstructed signals a novel fraud pattern not seen during training.

Probabilistic approaches estimate the probability density function of normal data and flag low-density regions as novel. Common techniques:

• Kernel Density Estimation (KDE) : Non-parametric density estimation where anomaly scores are inversely proportional to estimated density
• Gaussian Mixture Models (GMM) : Models normal data as a mixture of Gaussian distributions
• Normalizing Flows: Transforms simple distributions into complex ones through invertible mappings for exact likelihood computation
• Extreme Value Theory (EVT) : Models the tail of the anomaly score distribution to set mathematically rigorous thresholds

Setting the decision threshold is critical and challenging since only normal data is available during training. Approaches include:

• Extreme Value Theory: Fit a Generalized Pareto Distribution to the tail of anomaly scores on normal data to control false positive rate
• Percentile-based: Set threshold at the 95th or 99th percentile of anomaly scores on a held-out normal validation set
• Dynamic Thresholding: Adapt thresholds based on rolling statistics to account for concept drift and seasonality
• Contamination factor: Estimate the expected proportion of novel instances in production and calibrate accordingly

The foundational kernel method for novelty detection. A One-Class SVM learns a decision boundary that encapsulates the normal training data in a high-dimensional feature space, maximizing the margin from the origin. New transactions falling outside this boundary are flagged as novel.

• Uses the nu parameter to control the upper bound on outlier fraction
• Kernel trick (RBF) handles non-linear normality boundaries
• Effective when the normal class is tightly clustered

A neural extension of Support Vector Data Description that trains a network to map normal data into a minimal hypersphere in latent space. The distance from the sphere's center becomes the anomaly score.

• Jointly learns representations and the enclosing boundary
• Outperforms two-stage methods (autoencoder + separate scoring)
• Vulnerable to hypersphere collapse without architectural constraints

The broader task of identifying inputs that are semantically or statistically different from the training distribution. In fraud, this means detecting novel attack vectors never seen during training.

• Relies on softmax confidence, energy scores, or density estimation
• Critical for catching zero-day fraud patterns
• Prevents overconfident misclassifications on unknown schemes

A probabilistic model assuming data is generated from a mixture of Gaussian distributions. Novelty is scored by the probability density at a point's location—low-density regions indicate anomalies.

• Captures multi-modal normality (e.g., distinct customer segments)
• Provides calibrated likelihood scores for threshold setting
• Expectation-Maximization fits the model to clean data

A non-parametric method that estimates the probability density function by placing a kernel on each training point. Anomaly scores are inversely proportional to the estimated density.

• Makes no assumptions about the underlying data distribution
• Bandwidth parameter critically controls smoothness
• Computationally expensive at inference time for large datasets

A generative model that transforms a simple base distribution into a complex one through a sequence of invertible, differentiable mappings. Enables exact likelihood computation for novelty scoring.

• Provides tractable density estimation unlike VAEs
• Change-of-variables formula yields precise log-likelihoods
• Useful when the normal data distribution is highly non-Gaussian