Kullback-Leibler (KL) Divergence

KL divergence is directional and not a true distance metric: D_KL(P || Q) ≠ D_KL(Q || P).

• Forward KL (P || Q): Mode-covering behavior, penalizes Q where P has mass but Q does not
• Reverse KL (Q || P): Mode-seeking behavior, penalizes Q where Q has mass but P does not

In modulation classification, choosing the direction affects whether the classifier is conservative (avoiding false positives) or aggressive (capturing all signal variants).

Gibbs' inequality guarantees D_KL(P || Q) ≥ 0 for all distributions P and Q.

• Equality holds if and only if P = Q almost everywhere
• This property makes KL divergence a valid measure of discriminability between modulation hypotheses
• The minimum value of zero indicates that two signal models are statistically indistinguishable given the observed data

Minimizing D_KL(P_data || Q_model) is equivalent to maximizing the expected log-likelihood of the data under the model.

• The log-likelihood function is the empirical estimate of the negative cross-entropy
• In ALRT and GLRT classifiers, the likelihood ratio test statistic directly approximates the KL divergence between competing modulation hypotheses
• This connection bridges information theory and statistical decision theory

For independent random variables, KL divergence decomposes additively:

D_KL(P(x,y) || Q(x,y)) = D_KL(P(x) || Q(x)) + D_KL(P(y) || Q(y))

• Critical for analyzing sequential observations in SPRT-based classifiers
• Enables decomposition of multi-dimensional signal features into independent components
• Simplifies the calculation of discriminability bounds for IID channel models like AWGN

KL divergence is jointly convex in both arguments (P, Q).

• For fixed P, D_KL(P || Q) is convex in Q
• This property guarantees that optimization problems involving KL divergence have unique global minima
• Used in EM algorithms for blind modulation parameter estimation, where the auxiliary function is a convex KL divergence bound

KL divergence provides fundamental lower bounds on classifier performance:

• Stein's Lemma: The best achievable error exponent in hypothesis testing equals the KL divergence between the two distributions
• Sanov's Theorem: The probability of rare events decays exponentially with KL divergence
• These bounds define the theoretical limits of modulation recognition accuracy under finite sample constraints

The natural logarithm of the likelihood function, transforming products of conditional densities into sums for numerical stability. In modulation classification, the log-likelihood ratio between hypotheses is directly proportional to the KL divergence between the empirical data distribution and each candidate model. Computational necessity for avoiding floating-point underflow when processing long signal sequences.

A matrix measuring the amount of information an observable random variable carries about an unknown parameter. The KL divergence between a distribution and a locally perturbed version is asymptotically approximated by a quadratic form involving the FIM. This connects KL divergence to the Cramér-Rao Lower Bound, defining the ultimate precision limits for any unbiased modulation parameter estimator.

An information-theoretic model selection metric balancing goodness-of-fit with complexity by penalizing the log-likelihood with the number of estimated parameters. AIC is derived from minimizing the expected KL divergence between the true data-generating process and the fitted model. In modulation classification, AIC provides a principled method for selecting among candidate signal models when the true modulation order is unknown.

A model selection criterion similar to AIC but with a stronger complexity penalty scaling with the logarithm of sample size. BIC approximates the Bayes factor and is derived from asymptotic Bayesian arguments rather than direct KL divergence minimization. For large sample sizes common in signal processing, BIC favors simpler modulation hypotheses, reducing overfitting in automated classification systems.

A principle equating model selection with data compression, selecting the hypothesis providing the shortest encoding of the observed signal and the model itself. MDL is intimately connected to KL divergence through Kolmogorov complexity and universal coding theory. In modulation recognition, MDL provides a unified framework for jointly estimating modulation type and order without requiring prior distributions.

A loss function measuring the dissimilarity between predicted probability distributions and true labels. Cross-entropy equals the sum of the entropy of the true distribution and the KL divergence from the predicted to the true distribution. When training deep learning modulation classifiers, minimizing cross-entropy is equivalent to minimizing the KL divergence between the model's output and the one-hot encoded ground truth.