Proper Scoring Rule

The mean squared error between the predicted probability vector and the one-hot true outcome. It is a strictly proper scoring rule that decomposes into refinement and calibration components.

• Formula: ( \frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{K} (p_{ij} - y_{ij})^2 )
• Range: [0, 1] for binary, where 0 is perfect
• Key Property: Penalizes large errors quadratically, making it less sensitive to extreme probabilities than log-loss
• Use Case: Preferred when you want to evaluate both discrimination and calibration simultaneously

The negative log-likelihood of the true class under the predicted distribution. It is a local proper scoring rule, meaning it only depends on the probability assigned to the observed outcome.

• Formula: ( -\sum_{i} y_i \log(p_i) )
• Key Property: Heavily penalizes confident misclassifications (p ≈ 0 for true class → loss → ∞)
• Sensitivity: Drives models toward extreme calibration at the tails
• Connection: Minimizing log-loss is equivalent to maximum likelihood estimation

A strictly proper scoring rule for distributional forecasts that generalizes the Brier score to continuous outcomes. It measures the integrated squared difference between the predicted CDF and the empirical step function.

• Formula: ( \int_{-\infty}^{\infty} (F(y) - \mathbb{1}{y \ge y{true}})^2 dy )
• Key Advantage: Evaluates the full predictive distribution, not just point estimates
• Use Case: Standard metric in probabilistic weather forecasting and regression with uncertainty
• Property: Reduces to the Mean Absolute Error when predictions are deterministic

The multi-category generalization of the Brier score for ordinal or categorical forecasts. It sums the squared errors of the cumulative probabilities across all outcome thresholds.

• Formula: ( \frac{1}{K-1} \sum_{k=1}^{K} (\sum_{j=1}^{k} p_j - \sum_{j=1}^{k} y_j)^2 )
• Key Property: Penalizes predictions that are far from the true category more than those that are close
• Use Case: Evaluating ordinal classification where distance between categories matters (e.g., ratings, disease stages)
• Relationship: RPS collapses to the Brier score when K=2

A proper scoring rule for evaluating probabilistic forecasts that are summarized by their first two moments (mean and variance). It rewards forecast distributions that correctly specify both location and spread.

• Formula: ( \frac{(y - \mu)^2}{\sigma^2} + \log(\sigma^2) )
• Key Property: Only requires the forecaster to report a mean and variance, not a full distribution
• Use Case: Evaluating heteroscedastic regression models that output predictive uncertainty
• Advantage: Computationally simpler than CRPS while still rewarding honest variance estimates

A strictly proper multivariate generalization of the CRPS that evaluates probabilistic forecasts for vector-valued outcomes. It is based on the energy distance between the predicted distribution and the Dirac delta at the observation.

• Formula: ( \mathbb{E}[||X - y||] - \frac{1}{2}\mathbb{E}[||X - X'||] ) where X, X' are independent draws from the forecast distribution
• Key Property: Sensitive to both marginal calibration and dependence structure (correlation)
• Use Case: Evaluating spatial weather forecasts, multivariate time series, or any joint prediction task
• Computation: Requires Monte Carlo approximation via samples from the predictive distribution