Kernel Density Estimation (KDE)

Unlike Gaussian Mixture Models, KDE makes no assumptions about the underlying data distribution. It constructs a density estimate directly from observed transaction features, allowing it to model complex, multi-modal spending patterns that are characteristic of diverse customer bases.

• Adapts to arbitrary shapes: Captures non-Gaussian, skewed, and heavy-tailed distributions common in financial data.
• Data-driven: The final density function is entirely determined by the training samples, not a predefined formula.
• Contrast: Parametric methods like GMMs can fail when the true distribution of legitimate transactions deviates from a mixture of bell curves.

The bandwidth (h) parameter controls the smoothness of the estimated density and is the single most critical tuning decision. A small bandwidth creates a spiky, overfit density that can flag too many false positives, while a large bandwidth oversmooths and masks subtle fraud clusters.

• Small h: High variance, captures noise, risks overfitting to legitimate transactions.
• Large h: High bias, may merge a sparse fraud cluster into a nearby dense region of normal activity.
• Optimization: Cross-validation or Silverman's rule-of-thumb are standard starting points for bandwidth selection.

Anomaly scores are inversely proportional to the estimated probability density. A transaction in a sparse region of the feature space receives a low density value and a high anomaly score, directly quantifying how 'unusual' it is relative to historical norms.

• Score = -log(p(x)): A common transformation that maps low density to a high, unbounded anomaly score.
• Thresholding: A density cutoff is set on a validation set to control the expected false positive rate.
• Interpretability: The score directly answers 'how rare is this combination of features?' which is intuitive for fraud analysts.

The kernel function determines the shape of the influence each data point exerts on the density estimate. The Gaussian kernel is the most common choice for its smoothness and infinite support, but other kernels offer computational advantages.

• Gaussian Kernel: Smooth, differentiable, and mathematically convenient. The default for most applications.
• Epanechnikov Kernel: Optimal in a mean squared error sense and has finite support, which can speed up computation.
• Tophat/Uniform Kernel: Simple but produces discontinuous density estimates, rarely used in practice for fraud detection.

KDE performance degrades exponentially as the number of feature dimensions increases. In high-dimensional spaces, all points become equidistant, and density estimates lose discriminative power. This is a critical limitation for raw transaction data with dozens of features.

• Mitigation: Apply dimensionality reduction (PCA, autoencoder latent space) before KDE.
• Practical limit: KDE is most effective in 2 to 6 dimensions.
• Alternative: For high-dimensional anomaly detection, consider Isolation Forest or Deep SVDD, which do not rely on explicit density estimation in the full feature space.

A naive KDE evaluation requires computing the distance from a query point to every point in the training set, resulting in O(N) complexity per score. For real-time fraud scoring with millions of historical transactions, this is prohibitive without optimization.

• KD-Trees / Ball Trees: Approximate nearest-neighbor structures reduce query time to O(log N) for low-dimensional data.
• Binning / FFT: Fast Fourier Transform-based convolution can compute density on a grid in O(M log M) time.
• Sampling: Using a representative subsample of normal transactions as the reference set trades accuracy for speed.

A density-based algorithm that computes the local density deviation of a data point relative to its k-nearest neighbors. Unlike global methods, LOF identifies points that have a substantially lower density than their neighbors, making it highly effective for detecting local anomalies in heterogeneous financial transaction clusters where fraud patterns vary by region or customer segment.

• Anomaly Score: Ratio of the average local reachability density of a point's neighbors to its own local reachability density.
• Key Advantage: Adapts to varying cluster densities, flagging a moderate outlier in a sparse region while ignoring a similar point in a dense region.

A probabilistic model that assumes all data points are generated from a mixture of a finite number of Gaussian distributions. Anomalies are identified as points residing in low-density regions of the fitted probability density function. In fraud detection, GMMs model the multi-modal nature of normal behavior—such as distinct spending patterns for retail, travel, and online transactions—without requiring labeled fraud data.

• Output: A log-likelihood score for each transaction; points below a threshold are anomalous.
• Relationship to KDE: Both are non-parametric density estimators, but GMM explicitly models the data as a mixture of components, while KDE places a kernel on every data point.

A density-based spatial clustering algorithm that groups together points that are closely packed together, marking points that lie alone in low-density regions as noise or anomalies. DBSCAN requires two parameters: epsilon (the neighborhood radius) and minPts (the minimum number of points to form a dense region).

• Fraud Application: Identifies isolated transactions that do not belong to any legitimate behavioral cluster, such as a single high-value wire transfer from a new beneficiary.
• Contrast with KDE: DBSCAN provides a hard binary classification (core, border, noise), while KDE outputs a continuous density score for nuanced risk ranking.

An efficient unsupervised anomaly detection algorithm that creates a univariate histogram for each feature and scores an instance by the product of the inverse height of the bins it occupies. HBOS assumes feature independence, enabling extremely fast computation on high-dimensional transaction data.

• Scoring: A transaction falling into low-frequency bins across multiple features (e.g., unusual amount, rare merchant category, odd hour) receives a high anomaly score.
• Trade-off: Speed and scalability come at the cost of ignoring feature interactions, which KDE captures through its multivariate kernel function.

A statistical branch focused on modeling the tails of distributions. In anomaly detection, EVT is used to set mathematically rigorous thresholds by fitting a Generalized Pareto Distribution to the extreme values of anomaly scores, providing precise false positive control.

• Application: Instead of using an arbitrary percentile cutoff on KDE density scores, EVT models the tail of the score distribution to determine a threshold with a guaranteed probability of false alarm.
• Complementary Use: KDE estimates the full density; EVT refines the decision boundary in the critical low-density tail region where fraud resides.

A generative model that transforms a simple probability distribution (e.g., Gaussian) into a complex one through a sequence of invertible and differentiable mappings. Unlike KDE, which is a memory-based method requiring all training data at inference time, normalizing flows learn a parametric density model that allows for exact likelihood computation and efficient sampling.

• Anomaly Detection: Directly computes the negative log-likelihood of a transaction under the learned distribution as the anomaly score.
• Advantage over KDE: Scales to high-dimensional feature spaces without the curse of dimensionality that plagues kernel-based density estimators.