Negative Log-Likelihood (NLL)

NLL is a strictly proper scoring rule, meaning its expected value is uniquely minimized when the model outputs the true conditional class probabilities. This property encourages honest probability estimates rather than merely correct classifications. A model cannot 'game' NLL by being overconfident on easy examples while hedging on difficult ones—the optimal strategy is always to report the true predictive distribution. This contrasts with improper metrics like accuracy, which can be maximized by miscalibrated models.

NLL imposes a severe asymmetric penalty on confident misclassifications. Consider a binary case where the true label is class 1:

• High-confidence correct: p=0.99 → NLL = -log(0.99) ≈ 0.01 (minimal penalty)
• High-confidence wrong: p=0.01 → NLL = -log(0.01) ≈ 4.61 (extreme penalty)
• Uncertain prediction: p=0.50 → NLL = -log(0.50) ≈ 0.69 (moderate penalty) This logarithmic scaling means the penalty approaches infinity as the predicted probability for the true class approaches zero, making NLL highly sensitive to catastrophic errors.

For classification tasks with one-hot encoded targets, NLL is mathematically equivalent to categorical cross-entropy. Given a true distribution q (one-hot) and predicted distribution p, cross-entropy H(q,p) = -Σ q(x)log(p(x)) reduces to -log(p_true_class). This equivalence means minimizing NLL is identical to minimizing the Kullback-Leibler divergence between the empirical data distribution and the model's predicted distribution, making it the maximum likelihood estimator under a categorical likelihood assumption.

The expected NLL can be decomposed into two interpretable components:

• Calibration loss: Measures how well predicted probabilities match observed frequencies across confidence bins. A perfectly calibrated model has zero calibration loss.
• Refinement loss: Measures the inherent discriminative power of the model—how well it separates classes regardless of probability calibration. This is bounded by the dataset's intrinsic uncertainty. This decomposition reveals that NLL simultaneously optimizes for both sharpness (concentrated predictions) and calibration (accurate probabilities), unlike metrics such as Brier Score which weight these components differently.

The gradient of NLL with respect to the logits exhibits a critical property for optimization: ∂NLL/∂z_i = p_i - y_i, where p_i is the softmax probability and y_i is the one-hot target. This elegant form means:

• The gradient signal is strongest when the model is confidently wrong
• The gradient vanishes when the model is confidently correct
• The gradient naturally implements hard example mining, focusing parameter updates on misclassified or uncertain examples This property makes NLL particularly effective with stochastic gradient descent, as it automatically balances the learning signal across easy and hard examples.

NLL serves as a summary statistic for calibration quality that complements visual tools like reliability diagrams. While Expected Calibration Error (ECE) only measures miscalibration within discrete bins, NLL captures the full continuous probability spectrum. A model with low ECE but high NLL indicates diffuse predictions (poor refinement)—the model is calibrated but unconfident. Conversely, low NLL requires both good calibration and sharp, discriminative predictions. This makes NLL a more comprehensive single-number metric for probabilistic model quality than binning-based alternatives.

The primary empirical metric for measuring model calibration. ECE partitions predictions into M equally-spaced confidence bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin.

• Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|
• A perfectly calibrated model achieves an ECE of 0.
• Lower ECE indicates more reliable probability estimates.

A post-hoc calibration method that uses a single scalar parameter T to soften or sharpen the output softmax distribution. T > 1 softens probabilities, reducing overconfidence; T < 1 sharpens them.

• Optimized on a held-out validation set using NLL.
• Does not alter the model's accuracy or ranking—only the confidence scores.
• The simplest and most effective modern calibration technique for neural networks.

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and actual binary outcomes. It decomposes into refinement and calibration components.

• Formula: BS = (1/N) Σ (p_i - y_i)²
• Unlike NLL, the Brier Score is bounded between 0 and 1.
• Penalizes both overconfidence and underconfidence symmetrically.

A loss function that is minimized uniquely when the predicted probability distribution matches the true data-generating distribution. This property encourages honest and calibrated forecasts.

• Negative Log-Likelihood (NLL) is the canonical strictly proper scoring rule.
• The Brier Score is also strictly proper.
• Improper rules incentivize the forecaster to lie about their true beliefs.

A visual diagnostic tool that plots predicted confidence against observed accuracy across discrete bins. A perfectly calibrated model follows the identity diagonal (y = x).

• Bars above the diagonal indicate underconfidence.
• Bars below the diagonal indicate overconfidence.
• The gap between the curve and the diagonal is the visual representation of ECE.

Aleatoric uncertainty is the irreducible noise inherent in the data (e.g., sensor error). Epistemic uncertainty is the reducible model uncertainty due to lack of knowledge.

• Epistemic uncertainty is high in out-of-distribution (OOD) regions.
• Collecting more data reduces epistemic, but not aleatoric, uncertainty.
• Calibration primarily addresses the quality of the predicted distribution, while uncertainty quantification decomposes its source.