KL Divergence

KL divergence is not symmetric: (D_{KL}(P \parallel Q) \neq D_{KL}(Q \parallel P)). This is a core property, not a flaw. The order of the distributions matters:

• Forward KL ((P \parallel Q)): Measures the cost of using distribution Q to approximate the true distribution P. It is mode-covering; Q must assign probability mass everywhere P does, often leading to broad approximations.
• Reverse KL ((Q \parallel P)): Measures the cost of using P to approximate Q. It is mode-seeking; it allows Q to focus on a single mode of P, potentially ignoring others. In compression analysis, (P) is typically the original model's output distribution, and (Q) is the compressed model's, making (D_{KL}(P \parallel Q)) the standard measure of fidelity loss.

KL divergence is always non-negative: (D_{KL}(P \parallel Q) \geq 0). It equals zero if and only if the two distributions P and Q are identical almost everywhere. This property makes it a useful divergence (though not a formal distance metric).

• Implication for Compression: A KL divergence of zero indicates perfect fidelity; the compressed model's output distribution is statistically indistinguishable from the original's. Any positive value quantifies the information loss. This provides a clear, bounded-from-below metric for optimization: the goal is to minimize KL divergence toward zero.

KL divergence has a precise meaning in information theory: it measures the expected number of extra bits required to encode samples from the true distribution P using a code optimized for distribution Q. Formally, (D_{KL}(P \parallel Q) = \mathbb{E}_{x \sim P}[\log \frac{P(x)}{Q(x)}]).

• Cross-Entropy & Entropy: It can be expressed as (D_{KL}(P \parallel Q) = H(P, Q) - H(P)), where (H(P, Q)) is the cross-entropy and (H(P)) is the entropy of P.
• In Model Context: The 'extra bits' represent the surprisal or inefficiency incurred when the compressed model (Q) is used to explain data generated under the original model's (P) behavior. Minimizing KL divergence is equivalent to minimizing this coding inefficiency.

KL divergence is undefined if Q(x) = 0 for any x where P(x) > 0. This is because the term (\log \frac{P(x)}{Q(x)}) becomes infinite. This property enforces a hard constraint: the approximating distribution Q must assign non-zero probability to all events that P considers possible.

• Practical Impact in ML: When calculating KL divergence between model output distributions (e.g., softmax outputs), this is rarely an issue as outputs are usually smoothed. However, it necessitates careful numerical implementation (e.g., adding a tiny epsilon) when dealing with empirical distributions or sparse outputs to avoid infinite values.

In on-device compression, KL divergence serves as a direct measure of model fidelity, often more sensitive than top-1 accuracy for detecting distributional shifts.

• Primary Metric: It quantifies the output divergence between a golden model and its quantized/pruned variant.
• Comparison to Accuracy: A model may maintain similar top-1 accuracy but show high KL divergence, indicating its confidence scores across all classes have degraded—a sign of reduced calibration or robustness.
• Application: Used during calibration for post-training quantization to set activation ranges and in fine-tuning to recover accuracy by minimizing KL loss alongside task loss.

KL divergence is one of several f-divergences and relates to other statistical measures used in ML evaluation.

• Cross-Entropy: As noted, KL = Cross-Entropy - Entropy. During training, minimizing cross-entropy loss is equivalent to minimizing KL divergence between the true label distribution and the predicted distribution.
• Total Variation Distance & JS Divergence: These are other measures of distribution difference. Jensen-Shannon (JS) divergence is a symmetrized and smoothed version of KL divergence.
• Cosine Similarity: While cosine similarity measures angular alignment between vectors (like embeddings), KL divergence measures probabilistic discrepancy. For output distributions, KL provides a more statistically grounded measure of difference than cosine similarity.

The numerical discrepancy introduced when converting continuous floating-point values to discrete integer representations. This is the primary source of accuracy degradation in post-training quantization. The error is often modeled as additive noise that perturbs weights and activations. Minimizing this error is the goal of calibration and quantization-aware training.

The degree to which a compressed model's outputs and behaviors match those of the original, uncompressed golden model. It is a broader concept than simple task accuracy. Fidelity is measured using statistical metrics like KL divergence and cosine similarity between output distributions or internal activations. High fidelity ensures the compressed model 'reasons' similarly to the original.

A systematic evaluation to determine which components of a neural network are most critical for preserving accuracy. This involves:

• Measuring layer-wise sensitivity to compression.
• Identifying channels or parameters whose removal causes significant accuracy drop.
• The results directly inform mixed-precision quantization and structured pruning strategies, allowing aggressive compression in robust areas while protecting sensitive ones.

A small, representative set of unlabeled data used during post-training quantization. Its purpose is to estimate the dynamic range (min/max values) of model activations across different layers. This range is used to set the scaling factors and zero-points for integer conversion. A poor calibration dataset leads to suboptimal quantization parameters and increased quantization error.

In compression-accuracy analysis, the set of optimal points on a tradeoff curve. A point is on the Pareto frontier if no other compressed model configuration can improve one metric (e.g., accuracy) without worsening another (e.g., model size or latency). Engineering decisions aim to select operating points on this frontier that meet specific degradation thresholds.

The process of regaining performance lost during compression. This is not measured by KL divergence but is the ultimate engineering goal. Key techniques include:

• Fine-tuning after compression: Additional training of the compressed model on task data.
• Quantization-aware training (QAT): Simulating quantization noise during training to make the model robust.
• These methods aim to move a model's operating point closer to the Pareto frontier.