Neyman-Pearson Criterion

The Neyman-Pearson criterion solves a constrained maximization problem: maximize the probability of detection (P_D) subject to an upper bound on the probability of false alarm (P_FA).

• The constraint P_FA ≤ α defines the significance level or test size
• No prior probabilities or misclassification costs are required
• The solution is the likelihood ratio test (LRT) with a threshold chosen to satisfy the constraint
• This contrasts with Bayesian approaches that minimize average risk using priors and costs

The optimal decision rule under the Neyman-Pearson criterion always takes the form of a likelihood ratio test:

Λ(x) = p(x|H₁) / p(x|H₀) > η

• Λ(x) is the test statistic computed from observed data
• p(x|Hᵢ) are the conditional probability densities under each hypothesis
• η is the detection threshold determined by the P_FA constraint
• The test is uniformly most powerful (UMP) when testing simple hypotheses
• For composite hypotheses with unknown parameters, extensions like the GLRT are required

The critical threshold η is not arbitrary—it is uniquely determined by the specified false alarm probability:

P_FA = ∫_{x: Λ(x) > η} p(x|H₀) dx = α

• Requires knowledge of the null distribution of the test statistic
• For additive white Gaussian noise (AWGN) channels, the distribution is often tractable
• The relationship between η and P_FA is monotonic: higher thresholds reduce both P_FA and P_D
• Receiver operating characteristic (ROC) curves visualize this trade-off across all possible thresholds

A defining advantage of the Neyman-Pearson approach is its objectivity—it requires neither prior probabilities nor cost assignments.

• Bayesian methods demand P(H₀) and P(H₁), which may be unknown or contested
• Minimax criteria require a cost matrix Cᵢⱼ for each decision outcome
• Neyman-Pearson only requires specifying an acceptable false alarm rate α
• This makes it ideal for spectrum sensing and signal detection where priors are unavailable
• The trade-off: it does not guarantee minimum overall error probability, only optimal P_D for the chosen P_FA

In automatic modulation classification (AMC), the Neyman-Pearson criterion extends naturally to multi-class problems through pairwise or hierarchical testing:

• Each modulation candidate forms a composite hypothesis with unknown parameters (phase offset, timing, amplitude)
• The Generalized Likelihood Ratio Test (GLRT) replaces unknown parameters with their maximum likelihood estimates
• Average Likelihood Ratio Test (ALRT) treats parameters as random with known distributions
• Hybrid Likelihood Ratio Test (HLRT) combines both approaches for practical performance
• The criterion ensures a constant false alarm rate (CFAR) property across varying noise conditions

The Neyman-Pearson criterion is intrinsically linked to receiver operating characteristic (ROC) curves, which plot P_D against P_FA as the threshold varies.

• Each point on the ROC curve corresponds to a specific threshold η
• The Neyman-Pearson optimal test achieves the highest P_D for any given P_FA
• The area under the ROC curve (AUC) measures overall discriminability between hypotheses
• For modulation classifiers, ROC curves enable operating point selection based on mission requirements
• The Kullback-Leibler divergence between H₀ and H₁ determines the fundamental ROC shape

The ROC curve is the canonical visualization of the Neyman-Pearson criterion's performance trade-off. It plots the True Positive Rate (Probability of Detection) against the False Positive Rate (Probability of False Alarm) as the decision threshold varies.

• The Neyman-Pearson optimal detector corresponds to a single point on this curve for a fixed false alarm constraint.
• The Area Under the Curve (AUC) quantifies the classifier's overall discriminability between two modulation hypotheses.
• In signal classification, the ROC directly illustrates how increasing sensitivity to a target modulation inevitably increases susceptibility to false alarms from noise or other schemes.

CFAR is an adaptive thresholding technique that operationalizes the Neyman-Pearson principle in dynamic noise environments. It maintains a fixed Probability of False Alarm (P_fa) by continuously estimating the local noise floor from surrounding reference cells.

• Cell-Averaging CFAR (CA-CFAR) computes the mean noise power from adjacent range or frequency bins.
• Ordered-Statistic CFAR (OS-CFAR) selects the k-th ordered sample, providing robustness against interfering signals in the reference window.
• Essential for automatic modulation classifiers operating in congested spectrum where background interference is non-stationary.

The Neyman-Pearson criterion is straightforward for simple hypotheses (fully known distributions). In modulation classification, parameters like carrier phase, timing offset, and channel coefficients are unknown, creating composite hypotheses.

• The Generalized Likelihood Ratio Test (GLRT) substitutes unknown parameters with their Maximum Likelihood Estimates before applying the Neyman-Pearson threshold.
• The Average Likelihood Ratio Test (ALRT) treats unknowns as random variables and integrates them out, preserving the optimality structure.
• Neither approach is uniformly most powerful; the choice depends on the availability of prior distributions for nuisance parameters.

The SPRT extends the Neyman-Pearson fixed-sample framework to a variable-sample setting. Instead of collecting a fixed block of N samples, observations are processed sequentially until sufficient evidence crosses a decision boundary.

• Two thresholds, A and B, are derived from the desired P_fa and P_miss (probability of missed detection).
• The test minimizes the Average Sample Number (ASN) required to reach a decision under both hypotheses.
• Highly relevant for real-time modulation classifiers where minimizing latency and processing time is critical for spectrum agility.

KL Divergence measures the information-theoretic distance between the probability distributions of two modulation hypotheses. It directly relates to the asymptotic performance of the Neyman-Pearson detector.

• Stein's Lemma states that the best achievable exponential decay rate of the miss probability, for a fixed false alarm probability, equals the KL divergence between the two distributions.
• A larger KL divergence implies easier discrimination and fewer samples needed to achieve a target performance level.
• Used in feature selection to identify signal representations that maximize the statistical separation between candidate modulation formats.

The confusion matrix is the practical, multi-class extension of the binary detection metrics governed by the Neyman-Pearson criterion. For an M-ary modulation classifier, it is an M×M table where:

• Diagonal entries represent correct classifications (detections) for each modulation type.
• Off-diagonal entries represent specific misclassification errors, analogous to false alarms between specific modulation pairs.
• From this matrix, per-class Precision, Recall, and F1-Score are derived, providing a granular view of classifier performance beyond aggregate accuracy.