Proper Scoring Rule

The Brier score is a proper scoring rule for binary and multi-class predictions, defined as the mean squared error between the predicted probability vector and the one-hot encoded true outcome. For binary classification with predicted probability (\hat{p}) and true outcome (y \in {0,1}), it is calculated as (BS = \frac{1}{N} \sum_{i=1}^{N} (\hat{p}_i - y_i)^2).

• Properties: It is strictly proper, meaning a forecaster minimizes their expected score only by reporting their true belief. It simultaneously measures calibration (alignment of confidence and accuracy) and refinement (sharpness of predictions).
• Interpretation: Scores range from 0 to 2, with 0 representing a perfect forecast. A lower score indicates better predictive performance.
• Use Case: The standard metric for evaluating the overall quality of probabilistic classifiers in weather forecasting and many machine learning benchmarks.

Negative Log-Likelihood (NLL), also known as the log score, is a strictly proper scoring rule that measures the quality of a probabilistic forecast by penalizing low probability assigned to the observed outcome. For a model predicting probability (\hat{p}(y|x)) for the true class (y), it is (NLL = -\log \hat{p}(y|x)).

• Properties: It is local, meaning the score depends only on the probability assigned to the actual outcome, not the entire distribution. It is the primary loss function used to train most modern probabilistic classifiers (e.g., via cross-entropy).
• Interpretation: A lower NLL indicates a better model. It is sensitive to extreme overconfidence; assigning near-zero probability to a correct event yields an extremely high (poor) penalty.
• Use Case: The foundational training objective and evaluation metric for classification models, from logistic regression to large neural networks.

The logarithmic score is the generalization of NLL to full predictive distributions, not just classification. For a probabilistic forecast represented by a density (f) and an observed outcome (y), the score is (S(f, y) = \log f(y)).

• Properties: It is the premier example of a strictly proper local scoring rule. Its propriety is foundational to information theory; maximizing the expected log score is equivalent to minimizing the Kullback-Leibler divergence between the true and forecast distributions.
• Interpretation: Higher scores (closer to zero from below) are better. It strongly discourages forecasts that are overprecise (too narrow) or that ignore plausible outcomes.
• Use Case: The standard for evaluating density forecasts in fields like econometrics, statistics, and Bayesian modeling, where predicting a full distribution is required.

The spherical scoring rule is a proper scoring rule that measures the cosine similarity between the predicted probability vector and the vector representing the true outcome. For a prediction (\mathbf{p} = (p_1, ..., p_k)) and true outcome index (j), the score is (S(\mathbf{p}, j) = \frac{p_j}{|\mathbf{p}|}), where (|\mathbf{p}|) is the Euclidean norm.

• Properties: It is bounded between 0 and 1, with 1 being a perfect prediction. Unlike the log score, it is less sensitive to extreme errors, providing a more 'gentle' penalty.
• Interpretation: Encourages forecasters to be honest but does not penalize incorrect low-probability assignments as severely as the logarithmic score. It emphasizes the relative ranking of probabilities.
• Use Case: Used in contexts where forecasters may be risk-averse to the extreme penalties of the log score, or where a bounded, interpretable score is preferred.

The Ranked Probability Score (RPS) is a strictly proper scoring rule for ordinal outcomes, where categories have a natural order (e.g., 'low', 'medium', 'high'). It generalizes the Brier score by penalizing forecasts less if probability mass is placed on categories close to the true outcome.

• Calculation: For cumulative forecast distribution (F) and cumulative true distribution (G), (RPS = \sum_{i=1}^{k} (F_i - G_i)^2). It measures the squared error between the cumulative distributions.
• Properties: It respects the ordering of categories. A forecast of (0.1, 0.8, 0.1) for a 'medium' outcome scores better than (0.8, 0.1, 0.1), as the former placed more mass on the adjacent correct category.
• Use Case: The standard metric for evaluating probabilistic forecasts of ordered categorical variables, such as in weather forecasting (e.g., precipitation categories) or survey analysis (e.g., Likert scale responses).

The Continuous Ranked Probability Score (CRPS) is the generalization of the Ranked Probability Score (RPS) to continuous real-valued variables. It is a strictly proper scoring rule that compares a full predictive cumulative distribution function (CDF) against the observed value.

• Definition: (CRPS(F, y) = \int_{-\infty}^{\infty} (F(x) - \mathbb{1}{x \geq y})^2 dx), where (F) is the predictive CDF and (y) is the observed scalar. It can be interpreted as the integral of the Brier score at all probability thresholds.
• Properties: It measures both calibration and sharpness of a predictive distribution. A lower CRPS is better. It reduces to the mean absolute error if the forecast is a deterministic point.
• Use Case: The dominant metric for evaluating probabilistic forecasts of continuous quantities, ubiquitous in fields like numerical weather prediction, energy load forecasting, and quantitative finance.

The Brier score is a proper scoring rule that measures the mean squared error between predicted probabilities and true binary outcomes. It simultaneously evaluates calibration (how well confidence matches accuracy) and refinement (the sharpness of the predictions). A lower score indicates better performance.

• Example: For a binary classifier predicting rain with 80% confidence, and it does rain, the Brier score contribution is (0.8 - 1)^2 = 0.04.

Negative Log-Likelihood (NLL) is a proper scoring rule that measures the quality of a model's probabilistic predictions by penalizing low probability assigned to the correct class. It is the standard loss function for training probabilistic classifiers and serves as a fundamental evaluation metric.

• Mechanism: NLL is calculated as -log(p(y_true | x)), where p is the model's predicted probability for the true label. Perfect confidence yields an NLL of 0, while incorrect, high-confidence predictions are heavily penalized.

Expected Calibration Error (ECE) is a scalar summary metric that quantifies miscalibration. It works by:

1. Binning predictions into M intervals based on predicted confidence (e.g., [0.0, 0.1], [0.1, 0.2]).
2. Calculating the absolute difference between the average confidence and the empirical accuracy within each bin.
3. Averaging these differences, weighted by the number of samples in each bin. A lower ECE indicates better calibration, with 0 representing perfect alignment.

Post-hoc calibration refers to techniques applied to a trained model's outputs—without retraining—to improve probability calibration. A separate calibration set is used to fit a simple mapping function. Common methods include:

• Temperature Scaling: Applies a single scalar to soften or sharpen logits.
• Platt Scaling: Fits a logistic regression model to the outputs of a binary classifier.
• Isotonic Regression: Fits a non-parametric, piecewise constant function. These methods correct for the overconfidence common in modern neural networks.

Conformal prediction is a distribution-free framework for generating statistically valid prediction sets (for classification) or intervals (for regression) with guaranteed coverage probability. It provides rigorous uncertainty quantification for any underlying model.

• Process: Uses a calibration set to calculate a conformity score, then determines a threshold to create prediction sets that will contain the true label with a user-specified probability (e.g., 90%). It is a powerful method for achieving risk-controlling model outputs.

Calibration-aware training integrates calibration objectives directly into the model training process, aiming to produce intrinsically well-calibrated models. This contrasts with post-hoc methods. Techniques include:

• Label Smoothing: Replaces hard 0/1 labels with smoothed targets (e.g., 0.9/0.1), preventing overconfidence.
• Focal Loss: Down-weights loss for well-classified examples, indirectly mitigating overconfidence on easy samples.
• Bayesian Methods: Treats model parameters as distributions, naturally capturing predictive uncertainty.