Deep Open Classification

Unlike a two-stage pipeline, Deep Open Classification learns a unified feature manifold where known classes form compact, separable clusters and unknown samples are pushed into a low-density rejection region. This is achieved by training a single deep network with a composite loss function that simultaneously minimizes intra-class distance for known modulations while maximizing the entropy or reducing the feature magnitude for outlier data. The architecture typically uses a convolutional backbone for IQ sample feature extraction, followed by a fully connected embedding layer that projects features into a hypersphere or Euclidean space where distance metrics directly correspond to class membership confidence.

This architectural strategy represents each known modulation class not by a forward prototype, but by a reciprocal point in the embedding space. The network learns to map known samples close to their corresponding reciprocal point while maximizing the distance for unknown signals. Classification is performed by finding the minimum distance to any reciprocal point; if this distance exceeds a learned threshold, the sample is rejected as unknown. This creates a natural open space risk boundary because the space far from all reciprocal points is explicitly modeled as the rejection region, avoiding the closed-world assumption of traditional SoftMax classifiers.

Replaces the standard SoftMax output layer with a Dirichlet distribution over class probabilities. Instead of producing a point estimate, the network outputs the parameters of a Dirichlet distribution, treating predictions as subjective opinions with quantifiable uncertainty. The total evidence mass is inversely proportional to uncertainty: known modulations accumulate high evidence for a single class, while unknown signals produce uniformly low evidence across all classes. This allows the model to output "I don't know" with a mathematically grounded uncertainty score derived from the Dirichlet concentration parameters, enabling a principled rejection mechanism.

A specialized training objective that explicitly shapes the model's response to unknown signals. For known modulation samples, the loss encourages low-entropy, peaked SoftMax distributions concentrated on the correct class. For unknown or background samples, the loss enforces a high-entropy, uniform distribution across all known classes. This dual behavior creates a stark separability: known signals produce confident, low-entropy predictions, while unknown signals produce flat, high-entropy outputs. A simple entropy threshold on the output distribution then serves as an effective novelty detector without requiring an auxiliary outlier dataset.

This architecture adds a magnitude penalty to the feature embedding layer. The loss function is designed to maximize the L2 norm of feature vectors for known modulation classes while driving the norm of unknown samples toward zero. This creates a distinct feature magnitude gap where known signals reside on the surface of a hypersphere and unknown signals collapse near the origin. A single scalar threshold on the feature norm cleanly separates the two populations. This approach is particularly effective because it does not require modeling the distribution of unknowns, only their distance from the origin in the learned embedding space.

A training-time architectural augmentation that leverages an auxiliary dataset of diverse outlier examples to teach the network a tighter decision boundary. During training, batches are interleaved with real known modulation samples and synthetically generated or externally sourced outlier signals. The network is penalized for producing high-confidence predictions on these outliers, forcing it to learn a more conservative classification boundary. This regularization technique significantly improves out-of-distribution detection performance without modifying the inference architecture, making it a drop-in enhancement for existing deep modulation classifiers.

A regularization technique that dramatically improves out-of-distribution detection by training the model with an auxiliary dataset of diverse, non-task-related examples. By forcing the network to produce a uniform, high-entropy posterior over known classes for these outliers, the model learns a tighter, more conservative decision boundary. In signal classification, this means exposing the network to random noise or synthetically generated anomalous constellations during training.

A specialized loss function that creates a distinct separation in feature magnitude between known and unknown classes. The objective maximizes the feature vector norm for known modulation samples while minimizing it for unknown samples. This creates a clear, thresholdable boundary in the embedding space where the magnitude of the feature vector itself becomes a direct proxy for novelty, simplifying the rejection logic.

A framework that treats classification predictions as subjective opinions by placing a Dirichlet distribution over class probabilities instead of a point estimate. This allows the model to directly quantify epistemic uncertainty—the uncertainty arising from lack of knowledge. An input from an unknown modulation scheme will yield a flat Dirichlet distribution with high uncertainty, providing a principled statistical signal for rejection without requiring an explicit outlier class.

A classification strategy that represents each known class not by a central prototype, but by a reciprocal point in the embedding space. The model is trained so that known samples are close to their class's reciprocal point, while the maximum distance to all reciprocal points serves as the novelty score. This architecture explicitly models the open space risk by structuring the feature space to leave unbounded regions for unknown classes.

The continuous learning paradigm that extends beyond simple detection. An Open World system must not only reject unknown modulation types but also incrementally learn to recognize them when labeled data becomes available—without suffering catastrophic forgetting of previously learned classes. This is the operational reality for spectrum monitoring systems that must adapt to emerging waveforms over time.