Variational Autoencoder (VAE)

The core anomaly score in a VAE is the reconstruction probability, not the raw reconstruction error. This is computed using a Monte Carlo sampling technique. For a given input x, the VAE encodes it to a latent distribution, samples L latent vectors z, decodes each to a reconstruction distribution, and averages the probability of x under those distributions. This provides a more principled and stable metric than point-wise error, as it accounts for the variance of the latent space and the model's own uncertainty about the data.

Unlike a standard autoencoder, a VAE learns a smooth, continuous latent space by enforcing a prior distribution (typically a standard Gaussian) on the bottleneck layer. This is achieved through the KL divergence term in the loss function. The result is that similar normal transactions are mapped to overlapping regions, while anomalous inputs are forced into low-density regions of the prior, making them easily identifiable. This structure prevents the model from learning a trivial identity function.

Because a VAE models a distribution rather than memorizing a deterministic mapping, it is inherently more robust to the natural noise and variance present in normal financial transactions. A standard autoencoder might learn to perfectly reconstruct a slightly noisy but legitimate transaction, giving it a low error. A VAE, however, evaluates the probability of the data under a learned distribution, allowing it to accept minor, plausible variations as normal while flagging truly out-of-distribution anomalies.

The VAE's evidence lower bound (ELBO) can be decomposed to provide a more granular view of an anomaly. The score can be broken down into:

• Reconstruction term: How well the model can regenerate the input.
• KL Divergence term: How far the latent representation deviates from the prior. A fraudulent transaction might have a high reconstruction error, a high KL divergence (meaning it maps to an unusual part of the latent space), or both, offering explainability into why it was flagged.

As a generative model, a VAE can synthesize new, realistic samples of normal transaction data. This is invaluable for imbalanced classification problems. The generated samples can be used to augment the training set for downstream classifiers, or an analyst can use the model to visualize what a 'corrected' version of a fraudulent transaction would look like by encoding the anomaly and decoding a sample from its latent representation's nearest normal neighborhood.

A standard VAE can be extended to a Conditional VAE (CVAE) by conditioning both the encoder and decoder on known attributes, such as merchant category or transaction hour. This allows the model to learn a conditional distribution of normal behavior. An anomaly score is then computed as the reconstruction probability given the condition. A transaction that is normal for a grocery store would be highly anomalous if conditioned on a jewelry store, enabling highly contextual and precise fraud detection.

The VAE's primary anomaly scoring mechanism, which is more principled than standard reconstruction error. Instead of measuring the difference between input and output, it computes the probability of the input given the latent distribution. This is done by sampling multiple reconstructions from the latent space and averaging their likelihood, capturing the model's uncertainty. A low reconstruction probability indicates the data point lies in a low-density region of the learned normal manifold, making it a robust indicator of fraud.

The objective function maximized during VAE training. It balances two competing terms:

• Reconstruction Loss: How accurately the decoder reproduces the input from the latent sample.
• KL Divergence: How closely the learned latent distribution matches the prior (typically a standard Gaussian). Monitoring ELBO during training is critical; a poorly optimized ELBO leads to a latent space that does not accurately represent the density of normal transactions, degrading anomaly detection performance.

A deterministic precursor to the VAE. A standard autoencoder maps an input to a fixed point in a latent bottleneck and reconstructs it. Anomalies are flagged by a high reconstruction error. However, this approach lacks the VAE's probabilistic grounding. The latent space of a standard autoencoder can have undefined regions that decode to nonsensical outputs, whereas the VAE's regularized, continuous latent space provides a smooth manifold of normality, making anomaly boundaries more stable and interpretable.

The Kullback-Leibler Divergence term in the VAE loss acts as a regularizer, forcing the encoder's output distribution to be close to a standard normal prior. This prevents the model from memorizing training data by collapsing the variance to zero. For anomaly detection, this regularization ensures the latent space is continuous and smooth. A fraudulent transaction, when encoded, will be forced into a high-density region of the prior but will fail to reconstruct accurately, resulting in a low reconstruction probability.

An alternative deep anomaly detection method that trains a neural network to map all normal data into a minimal hypersphere in the latent space. The anomaly score is the distance from the sphere's center. Unlike a VAE, Deep SVDD does not have a decoder and cannot generate data. It focuses purely on a compact normality representation. VAE-based methods are often preferred when the data manifold is complex and multi-modal, as the Gaussian mixture in the latent space can capture more nuanced distributions than a single hypersphere.

Uses a Generative Adversarial Network to learn the manifold of normal data. To score an anomaly, it searches for a latent vector that generates a similar image via backpropagation, combining a residual loss (visual difference) and a discrimination loss (feature matching). Compared to VAEs, GAN-based methods can produce sharper generations but suffer from mode collapse and unstable training. VAEs provide a more stable, likelihood-based framework, which is often preferred in audited financial environments requiring consistent, reproducible anomaly scores.