Autoencoder Anomaly Detection

The core principle is that an autoencoder trained on normal signal data minimizes reconstruction error for known modulations. When presented with an unknown modulation scheme, the encoder compresses it into a latent space optimized for normal data, and the decoder produces a degraded reconstruction. The Mean Squared Error (MSE) between the input and reconstruction serves as a continuous anomaly score.

• A high reconstruction error indicates the input lies outside the learned data manifold
• A threshold is set on the error distribution of a validation set to define the rejection boundary
• This provides a non-parametric approach that makes no assumptions about the distribution of unknown classes

The information bottleneck in the autoencoder's middle layer is critical. By forcing the network to compress high-dimensional IQ samples through a narrow latent representation, it learns only the most salient features of normal data. Unknown modulations, possessing different statistical structures, cannot be efficiently encoded.

• The latent dimensionality is a crucial hyperparameter: too wide, and the network learns the identity function; too narrow, and it fails to reconstruct even normal data
• Variational Autoencoders (VAEs) impose a probabilistic prior on the latent space, making the bottleneck stochastic and improving novelty detection by regularizing the manifold

A binary classifier is created by applying a decision threshold to the reconstruction error. Selecting this threshold involves a trade-off between true positive rate (TPR) for known classes and false positive rate (FPR) for anomalies.

• The threshold is typically set using a held-out validation set of normal data, often at the 95th or 99th percentile of the error distribution
• Extreme Value Theory (EVT) can be applied to the tail of the error distribution to model the probability of extreme reconstruction errors, providing a statistically principled threshold
• Dynamic thresholds can adapt to varying channel conditions, such as changing signal-to-noise ratios

Standard autoencoders are vulnerable to adversarial perturbations—small, carefully crafted noise that causes a high reconstruction error for known classes, triggering false positives. Robust training techniques mitigate this.

• Adversarial training augments the training data with perturbed examples, forcing the autoencoder to learn a smoother reconstruction manifold
• Contractive autoencoders add a penalty term to the loss function that minimizes the Jacobian of the encoder, making the latent representation resistant to small input variations
• This is critical in electronic warfare contexts where signals may be intentionally jammed or spoofed

Autoencoder anomaly detection is often combined with a closed-set classifier to build a complete open set recognition system. The classifier identifies known modulation types, while the autoencoder flags inputs that are unlike any known class.

• The OpenMax layer can replace SoftMax by using the reconstruction error from a per-class autoencoder ensemble to recalibrate activation vectors
• Objectosphere loss can be used to jointly train a feature extractor and an autoencoder, maximizing feature magnitude for known classes while minimizing it for unknowns
• This hybrid approach reduces open space risk by explicitly modeling the boundary of known data

Autoencoders are well-suited for deployment on FPGAs and embedded systems due to their feed-forward architecture and deterministic inference time. Unlike generative adversarial networks, they require only a single forward pass.

• Post-training quantization reduces model size to 8-bit integers with minimal loss in reconstruction fidelity
• Pruning removes redundant neurons in the decoder, which typically has more parameters than the encoder
• Inference latency is constant and predictable, critical for real-time spectrum monitoring applications where decisions must be made within a single time slot

The fundamental anomaly score in autoencoder systems, calculated as the mean squared error (MSE) or mean absolute error (MAE) between the original input IQ samples and the decoder's output. A high reconstruction error indicates the input deviates from the learned manifold of normal modulation types. The threshold is typically set using a percentile of training errors or by fitting a statistical distribution to the error values.

The compressed latent space at the center of the autoencoder architecture where the encoder forces high-dimensional IQ data through a dimensionality reduction layer. This bottleneck learns a compact, salient representation of normal modulation schemes. The information bottleneck principle ensures that only the most statistically significant features of known signals are preserved, making novel modulation patterns inherently difficult to reconstruct.

A probabilistic variant that encodes inputs as mean and variance vectors parameterizing a Gaussian latent distribution, rather than a deterministic point. For anomaly detection, VAEs enable sampling-based reconstruction and provide a principled measure of novelty through the Kullback-Leibler divergence term. The reconstruction probability, which accounts for the variance of the output, often provides more robust anomaly scoring than raw reconstruction error.

An alternative to reconstruction-based methods that learns a minimum-volume hypersphere enclosing the latent representations of normal data. Anomalies are detected when a sample's embedding falls outside this learned boundary. Unlike autoencoders, Deep SVDD trains with a center-loss objective that explicitly minimizes the distance of all normal embeddings to a fixed center, creating a compact normality region in feature space.

A statistically-informed distance metric that accounts for the covariance structure of the learned latent representations. Rather than using Euclidean distance to a class prototype, the Mahalanobis distance normalizes by the feature distribution's spread, providing superior out-of-distribution detection. It is computed as: D(x) = sqrt((z - μ)ᵀ Σ⁻¹ (z - μ)) where z is the latent embedding, μ is the class mean, and Σ is the covariance matrix.

A regularization technique that injects auxiliary outlier data during autoencoder training to sharpen the reconstruction boundary. By forcing the model to produce high reconstruction error on known outliers while maintaining low error on in-distribution signals, the network learns a tighter decision boundary. This directly addresses the open space risk where unknown modulations might inadvertently map to low-error regions of the latent space.