Inferensys

Glossary

Open Set Recognition

A classification paradigm where the model must accurately identify known classes while simultaneously detecting and rejecting samples from unknown or novel classes not seen during training.
ML engineer managing model training cluster on laptop, GPU utilization visible, technical deep learning setup.
CLASSIFICATION PARADIGM

What is Open Set Recognition?

Open Set Recognition (OSR) is a classification paradigm where the model must accurately identify known classes while simultaneously detecting and rejecting samples from unknown or novel classes not seen during training.

Open Set Recognition fundamentally breaks the restrictive closed-set assumption of traditional classifiers. Instead of forcibly mapping every input to a known class, an OSR system learns a decision boundary that encloses known data while quantifying open space risk—the danger of labeling an unknown input as known. This is critical in dynamic spectrum environments where new modulation types constantly emerge.

The mechanism typically involves learning a compact representation of each known class, often using prototype learning or Extreme Value Theory to model the tail distribution of activation distances. At inference, a sample is rejected if its distance to the nearest known class exceeds a calibrated threshold, often derived from Mahalanobis distance or an entropic open-set loss that forces high-entropy predictions for unknowns.

BEYOND CLOSED-SET CLASSIFICATION

Key Features of Open Set Recognition

Open Set Recognition moves beyond the closed-set assumption to build classifiers that know what they don't know—accurately identifying known modulation types while rejecting unknown signals never seen during training.

01

OpenMax: Replacing SoftMax with Rejection

OpenMax replaces the standard SoftMax layer by fitting a Weibull distribution to the distance of correct classifications from their class mean using Extreme Value Theory. The recalibrated activation vector includes an explicit 'unknown' score, allowing the model to reject inputs that fall far from all known class representations. This is the foundational deep learning approach for open set recognition.

Weibull
Tail Distribution
EVT
Statistical Framework
02

Objectosphere Loss: Separating by Feature Magnitude

Objectosphere Loss creates a distinct separation in feature space by maximizing the feature norm for known samples while minimizing it for unknown samples. This dual-objective training produces a thresholdable rejection space where unknown signals naturally collapse toward the origin, making them easily distinguishable from high-magnitude known class embeddings.

Known
High Feature Norm
Unknown
Low Feature Norm
03

Reciprocal Point Learning

This strategy represents each known class by a reciprocal point in the embedding space rather than a centroid. Classification uses the maximum distance to these reciprocal points. Unknown samples are identified when they fall far from all reciprocal points, creating an open space risk-minimizing boundary that naturally rejects novel modulation schemes.

04

Entropic Open-Set Loss

This training objective forces the network to produce high-entropy, uniform probability distributions for unknown samples while maintaining low-entropy predictions for known classes. The entropy gap creates a clear separation signal: known modulations produce confident, peaked outputs, while novel signals generate flat, uncertain distributions that trigger rejection.

05

Outlier Exposure for Tighter Boundaries

Outlier Exposure is a regularization technique that trains the model with an auxiliary dataset of diverse outlier examples. By forcing the network to learn a tighter decision boundary around known classes, it dramatically improves out-of-distribution detection. In spectrum monitoring, this means exposing the classifier to varied noise patterns and interference signals during training.

06

Epistemic Uncertainty via Deep Ensembles

Deep Ensembles quantify epistemic uncertainty by training multiple neural networks with different random initializations. The variance of their predictions serves as a robust signal for detecting unknown inputs. When ensemble members disagree significantly on a modulation classification, the sample is likely from an unknown class, triggering rejection.

Variance
Uncertainty Signal
Ensemble
Detection Method
OPEN SET RECOGNITION

Frequently Asked Questions

Clear, technically precise answers to the most common questions about building classifiers that know what they don't know, enabling robust spectrum monitoring in dynamic electromagnetic environments.

Open Set Recognition (OSR) is a classification paradigm where a model must simultaneously accurately identify known classes and detect and reject samples from unknown or novel classes not seen during training. This fundamentally differs from traditional classification, which operates under the closed-set assumption—the premise that all test classes are identical to the training classes. In a closed-set system, a new, unseen modulation type will be forcibly misclassified into one of the known categories with high confidence, creating a silent failure. OSR introduces a rejection mechanism, allowing the model to output an 'unknown' or 'novel' label. This is critical in dynamic spectrum environments where new waveforms, jammers, or interference patterns constantly emerge. The core challenge is managing open space risk, the theoretical risk of labeling an unknown input as a known class, which is quantified as the relative measure of the feature space far from any known training data that is nonetheless classified as known.

DEPLOYMENT DOMAINS

Real-World Applications of Open Set Recognition

Open Set Recognition (OSR) moves beyond the closed-world assumption, enabling classifiers to operate safely in dynamic environments where unknown modulation schemes inevitably appear. These applications demonstrate how rejecting novelty is as critical as identifying known signals.

01

Cognitive Radio & Dynamic Spectrum Access

In congested electromagnetic environments, secondary users must identify spectrum holes without causing interference to primary incumbents. An OSR-based classifier identifies known standardized waveforms (LTE, 5G NR, Wi-Fi) while rejecting unknown or proprietary signals as a distinct 'unknown' class. This prevents a cognitive radio from mistakenly classifying a novel radar pulse as a vacant band, avoiding dangerous collisions. The system uses Entropic Open-Set Loss to force high uncertainty on unfamiliar emissions, triggering a safe fallback mode rather than a catastrophic misclassification.

< 1 ms
Decision Latency Target
99.9%
Unknown Detection Rate
02

Electronic Warfare & SIGINT

Signals intelligence (SIGINT) operators face an endless stream of adversarial and civilian emitters. A closed-set classifier fails silently when a new enemy waveform appears. OSR models, using Reciprocal Point Learning, map known threat modulations to compact embedding regions. Any signal falling outside these reciprocal boundaries is flagged as a novel threat emitter for immediate human analysis. This capability is critical for identifying low-probability-of-intercept (LPI) signals and adaptive frequency-hopping patterns that were absent from the training corpus.

100%
Novel Threat Capture
03

Spectrum Monitoring & Regulatory Enforcement

National regulatory bodies monitor vast frequency ranges to detect illegal or unlicensed transmissions. OSR automates this by learning the 'normal' RF background of a city. Using Mahalanobis Distance in a deep feature space, the system flags any transmission that deviates statistically from known commercial allocations. This allows an inspector to instantly identify a rogue transmitter using a non-standard modulation scheme, rather than manually scanning waterfalls. The system reduces false positives by calibrating epistemic uncertainty against varying urban noise floors.

04

Satellite Communication Ground Stations

Ground stations must autonomously demodulate telemetry from diverse, aging, and often undocumented satellite fleets. An OSR system trained on standard CCSDS protocols can lock onto known downlinks while rejecting unknown interference or spoofing attempts. By applying Temperature Scaling to output logits, the classifier provides a calibrated confidence score. If a signal from a newly launched adversary satellite is received, the low confidence triggers a raw IQ recording for forensic analysis rather than attempting a forced, erroneous demodulation.

05

Industrial IoT Interference Diagnosis

Factory floors contain a chaotic mix of Bluetooth, proprietary wireless sensors, and high-noise machinery. An OSR-based diagnostic tool learns the spectral signatures of authorized equipment. When a new jammer or a malfunctioning motor radiates an unknown electromagnetic signature, the system detects it as out-of-distribution using an Autoencoder reconstruction error threshold. This allows maintenance crews to pinpoint novel interference sources without needing a pre-labeled database of every possible fault mode.

06

Autonomous Vehicular V2X Security

Vehicle-to-Everything (V2X) communication relies on standardized IEEE 802.11p or C-V2X waveforms. An OSR module within the vehicle's security gateway verifies that incoming safety messages conform to known modulation profiles. Using Deep Ensembles, the variance of predictions across multiple models is measured. A high variance indicates an unknown or adversarial signal, potentially a spoofing attack using a non-standard waveform, which is immediately rejected to prevent the autonomous system from acting on malicious data.

CLASSIFICATION PARADIGM COMPARISON

Open Set vs. Closed Set Recognition

A technical comparison of the assumptions, capabilities, and failure modes of closed-set and open-set classification paradigms for signal identification.

FeatureClosed Set RecognitionOpen Set Recognition

Core Assumption

Test classes are identical to training classes

Test data may contain unknown or novel classes

Unknown Class Handling

Decision Boundary

Partitions entire feature space among known classes

Known classes bounded; remaining space labeled unknown

Open Space Risk

Unbounded (high risk of misclassifying unknowns)

Minimized via explicit rejection boundary

Output Space

K-dimensional probability vector (sums to 1)

K+1 dimensional output (K known + 1 unknown class)

Typical Activation

SoftMax

OpenMax, Entropic Open-Set, or Objectosphere loss

Deployment Suitability

Static, controlled environments

Dynamic spectrum environments with novel emitters

Failure Mode

Silent misclassification of novel signals

Explicit rejection with unknown flag

Prasad Kumkar

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.