Continuous Ranked Probability Score

A scoring rule is strictly proper if its expected value is uniquely maximized when the forecaster reports their true belief distribution. CRPS satisfies this property, meaning a forecaster cannot 'game' the metric by issuing a biased or overconfident prediction. The optimal strategy is always to report the true predictive distribution. This property is critical for supply chain applications where forecasters might otherwise be incentivized to inflate confidence to appear more accurate.

Unlike classification metrics that only penalize incorrect decisions, CRPS penalizes forecasts based on how far the predicted distribution is from the observed outcome. A forecast that places high probability mass near the true value receives a better score than one that is far off, even if neither assigns high probability to the exact outcome. This distance sensitivity makes CRPS ideal for inventory optimization, where a forecast error of 10 units is less costly than an error of 1000 units.

CRPS evaluates the entire predicted Cumulative Distribution Function (CDF) against the observed outcome, not just a point estimate or a single prediction interval. This means it simultaneously assesses:

• Calibration: Are the predicted probabilities statistically consistent with observed frequencies?
• Sharpness: How concentrated is the predictive distribution around the outcome? A forecast can only achieve a low CRPS by being both well-calibrated and sharp, preventing the 'hedging' behavior seen with interval-based metrics.

CRPS can be understood as a probabilistic generalization of the Mean Absolute Error (MAE). When a forecast collapses to a deterministic point prediction (a Dirac delta distribution), the CRPS reduces exactly to the absolute error between the prediction and the observation. This property provides an intuitive bridge: for a deterministic forecaster, CRPS equals MAE; for a probabilistic forecaster, CRPS rewards honest uncertainty quantification around that same central tendency.

The CRPS can be decomposed into interpretable components that reveal why a forecast is performing well or poorly:

• Reliability (Calibration): Measures whether predicted probabilities match observed frequencies.
• Resolution: Measures the forecaster's ability to issue different predictions when different outcomes occur.
• Uncertainty: The inherent variability in the observations themselves. This decomposition enables targeted model improvement, such as recalibrating a model that shows poor reliability without discarding its useful resolution capability.

CRPS has an equivalent formulation as the integral of the pinball loss over all quantile levels from 0 to 1. This relationship is practically significant: a model that outputs a set of predicted quantiles (e.g., the 5th, 25th, 50th, 75th, and 95th percentiles) can approximate CRPS by averaging the pinball loss across those quantiles. This connects CRPS directly to quantile regression methods and enables efficient computation without explicitly integrating the full CDF.

The mean squared error equivalent for probabilistic forecasts of binary events. While CRPS generalizes to continuous variables, the Brier Score assesses the accuracy of probability forecasts for discrete outcomes like 'Will stockout occur?'

• Ranges from 0 (perfect) to 1 (worst)
• Decomposable into reliability, resolution, and uncertainty components
• Essential for evaluating demand classification models that trigger replenishment alerts

A diagnostic tool that transforms observed outcomes using the forecasted CDF. If the probabilistic forecast is ideal, the resulting PIT values will be uniformly distributed on [0,1].

• A U-shaped PIT histogram indicates underdispersed forecasts (too narrow prediction intervals)
• A hump-shaped PIT histogram signals overdispersed forecasts (too wide intervals)
• Used alongside CRPS to visually diagnose calibration deficiencies in demand models

The mathematical property that ensures a forecaster's expected score is optimized by reporting their true belief. Strict propriety means the maximum expected reward occurs uniquely at the true distribution.

• CRPS is strictly proper, incentivizing honest uncertainty quantification
• Improper scores (like the absolute error of the median) encourage hedging or manipulation
• Essential for designing multi-agent forecasting markets where participants are rewarded for truthful predictions

The maxim that probabilistic forecasts should be as sharp as possible subject to being calibrated. Sharpness refers to the concentration of the predictive distribution—narrower prediction intervals are sharper.

• CRPS rewards both calibration and sharpness simultaneously
• A climatological average is perfectly calibrated but not sharp, yielding a poor CRPS
• The principle guides hyperparameter tuning in DeepAR and Temporal Fusion Transformer models to avoid overly conservative forecasts

A statistical hypothesis test that compares the predictive accuracy of two forecasting models using a time series of score differences. When applied to CRPS values, it determines if one model's probabilistic superiority is statistically significant.

• Accounts for autocorrelation in forecast errors
• Null hypothesis: both models have equal expected CRPS
• Critical for enterprise model selection when deploying new demand forecasting systems into production