Model Drift Detection

The most common form of drift, occurring when the statistical distribution of the input IQ data P(X) changes, while the conditional distribution of the label given the input P(Y|X) remains constant. In RFML, this manifests as a shift in the signal-to-noise ratio (SNR), the appearance of new interference types, or a change in the channel's delay spread.

• Example: A modulation classifier trained on high-SNR, line-of-sight (LOS) signals fails when deployed in a low-SNR, non-line-of-sight (NLOS) multipath environment.
• Detection Method: Use a two-sample statistical test like Maximum Mean Discrepancy (MMD) on the latent feature embeddings of a reference and a production data window.

A change in the distribution of the target classes P(Y) without a change in the likelihood P(X|Y). In an RF spectrum monitoring context, this happens when a previously rare emitter type suddenly becomes dominant, or a new operational mode is introduced.

• Example: A peacetime spectrum monitoring model encounters a sudden surge of frequency-hopping spread spectrum (FHSS) signals during a military exercise, a class that was severely underrepresented in the training set.
• Detection Method: Monitor the marginal distribution of the model's softmax output probabilities using a drift detector like the Kolmogorov-Smirnov (KS) test on the prediction confidence scores.

The most severe form of drift, where the fundamental relationship between the input and the target changes, meaning P(Y|X) is no longer stable. The same IQ sample now corresponds to a different true class. This often indicates a change in the physical hardware or transmission protocol.

• Example: A specific emitter identification (SEI) model trained to fingerprint a particular radar's power amplifier non-linearity begins to fail after the adversary replaces the amplifier hardware, altering the unique fingerprint.
• Detection Method: Requires ground-truth labels in production, making it the hardest to detect online. Performance metrics like accuracy or F1-score are calculated on a delayed labeled sample set and compared against a baseline.

A drift pattern specific to time-series RF data where the statistical properties of the channel evolve over time, even if the emitter behavior is static. This is driven by physical phenomena like Doppler spread from moving reflectors or channel aging in a mobile environment.

• Example: An autoencoder-based neural receiver trained for a static pedestrian channel experiences catastrophic decoding failure when the user enters a high-speed vehicle, drastically changing the Doppler profile.
• Detection Method: Track the reconstruction error of a variational autoencoder (VAE) trained on nominal channel conditions. A sustained increase in the reconstruction loss signals a departure from the learned temporal dynamics.

A deliberate, malicious form of concept drift where an attacker introduces carefully crafted adversarial perturbations to the input signal to force misclassification. Unlike natural drift, these perturbations are designed to be imperceptible to traditional statistical monitoring while maximizing model error.

• Example: An adversary transmits a waveform with a subtle, optimized perturbation layer that causes an automatic modulation classifier (AMC) to confuse a QPSK signal with a 16-QAM signal, evading detection.
• Detection Method: Use a separate outlier detector based on the local intrinsic dimensionality (LID) of the model's hidden layer activations, which is sensitive to the adversarial subspace.

A non-statistical form of drift caused by a failure in the signal processing pipeline rather than the environment. This includes sensor miscalibration, IQ imbalance, local oscillator (LO) leakage, or a change in the digital-to-analog converter's (DAC) sampling rate.

• Example: A temperature-induced DC offset in the receiver's analog front-end introduces a constant tone at the center frequency, corrupting all downstream IQ samples and causing the model to see a non-existent carrier.
• Detection Method: Implement data validation checks on raw IQ statistics (mean, variance, kurtosis) and spectral flatness before inference. A schema violation triggers an alert before the corrupted data reaches the model.

The fundamental phenomenon where the statistical properties of the target variable change over time. In RF systems, this occurs when the relationship between input IQ samples and their correct labels shifts.

• Virtual Drift: The decision boundary learned during training no longer matches the real-world distribution, even if the input data appears similar.
• Real Drift: A tangible change in the electromagnetic environment, such as a new interferer or a change in a transmitter's power amplifier behavior.
• Covariate Shift: A specific type where the input distribution P(X) changes, but the conditional label distribution P(Y|X) remains constant.

A change in the input data distribution P(X) that the model receives in production compared to the training data. This is the most commonly monitored signal for drift.

• Feature-Level Monitoring: Tracking the mean, variance, and histogram of specific signal features like Error Vector Magnitude (EVM) or Delay Spread.
• Raw IQ Monitoring: Directly comparing the distribution of raw in-phase and quadrature samples using statistical distance metrics.
• Causes: A new radio operating in the band, seasonal foliage changes affecting multipath, or hardware degradation in a receiver's low-noise amplifier.

A non-symmetric statistical distance metric that quantifies how one probability distribution diverges from a second, reference distribution. A cornerstone algorithm for drift quantification.

• Drift Score: A KL divergence value of 0 indicates identical distributions; increasing values signal growing drift.
• Application: Computed between the histogram of model outputs on a reference dataset and the histogram of outputs on live production data.
• Alternative Metrics: Jensen-Shannon Divergence (a symmetric variant) and Wasserstein Distance are often preferred for their stability and geometric interpretability.

A symmetric metric derived from information theory that measures the shift in a variable's distribution across discrete bins. Widely adopted in financial model governance and increasingly in ML operations.

• Interpretation: PSI < 0.1 indicates no significant shift; 0.1–0.25 suggests moderate drift; > 0.25 signals a major distribution change requiring model retraining.
• RF Application: Applied to binned signal features like Rician K-Factor or Doppler Spread estimates to detect environmental changes.
• Advantage: Provides a single, intuitive scalar value that non-ML stakeholders can use for governance decisions.

A model's intrinsic ability to recognize and flag input RF signals that belong to an unknown class or environment not represented in the training data. This is a proactive defense against silent misclassifications.

• Softmax Thresholding: Rejecting predictions where the maximum softmax probability falls below a calibrated threshold.
• Energy-Based Models: Using the Helmholtz free energy of a model's logits as a detection score, which often outperforms softmax confidence.
• RF Specificity: Critical for detecting novel jamming waveforms or unknown modulation schemes that a classifier has never seen.

A scalar metric that quantifies the mismatch between a model's reported confidence scores and its actual empirical accuracy. A perfectly calibrated model has an ECE of zero.

• Reliability Diagrams: Visual plots that bin predictions by confidence and compare them to observed accuracy, revealing overconfidence or underconfidence.
• Drift Correlation: A rising ECE over time is a strong indicator of concept drift, as the model becomes increasingly miscalibrated on new data.
• Mitigation: Temperature scaling, a post-hoc calibration method, can be applied to logits to reduce ECE without changing the model's accuracy ranking.