Continuous Ranked Probability Score (CRPS)

A scoring rule is strictly proper if the expected score is uniquely maximized (or minimized) when the forecaster reports their true belief distribution. For CRPS, the minimum expected value is achieved only when the predicted CDF matches the true data-generating distribution.

• No hedging: Forecasters cannot improve their expected score by reporting a distribution different from their true belief
• Unique optimum: Unlike improper rules, there is exactly one distribution that minimizes the expected CRPS
• Comparison: The Mean Absolute Error (MAE) is not a proper scoring rule for probabilistic forecasts, while CRPS is

CRPS can be decomposed into three interpretable components: uncertainty, reliability, and resolution. This decomposition reveals whether a poor score stems from miscalibration or lack of sharpness.

• Uncertainty: The inherent unpredictability of the observation, measured by the variance of the climatological distribution
• Reliability (Calibration): How well the predicted probabilities match observed frequencies across all probability levels
• Resolution (Sharpness): The ability to issue forecasts that differ from the climatological mean, concentrating probability mass tightly around the eventual observation

A perfectly calibrated but unsharp forecast (always predicting the climatological distribution) achieves zero resolution and a CRPS equal to the uncertainty component.

CRPS penalizes forecasts based on the absolute distance between the predicted CDF and the empirical observation. Unlike the logarithmic score, which can assign infinite penalties for zero-probability events, CRPS is robust to outliers and bounded in its sensitivity.

• Linear penalty growth: The penalty scales linearly with the distance between prediction and observation, not quadratically
• No infinite penalties: A forecast that assigns zero probability to the observed outcome receives a finite penalty proportional to the distance
• Unit consistency: CRPS is expressed in the same units as the target variable, making it directly interpretable for domain experts

This property makes CRPS particularly suitable for financial time series with heavy-tailed distributions where extreme events occur more frequently than Gaussian assumptions predict.

CRPS naturally evaluates ensemble forecasts without requiring a parametric distribution. Given a set of M ensemble members, the CRPS can be computed directly from the samples using the fair CRPS estimator.

• Sample-based formula: CRPS = (1/M) Σ|ŷᵢ - y| - (1/2M²) Σ|ŷᵢ - ŷⱼ|, where ŷᵢ are ensemble members and y is the observation
• No distributional assumption: Works with any collection of samples, including those from non-parametric models like quantile regression forests or deep generative models
• Fair estimator: The second term corrects for finite ensemble size, ensuring unbiased estimation even with small ensembles

This property makes CRPS the standard evaluation metric for probabilistic forecasting competitions and operational ensemble weather prediction systems.

CRPS is the integral of the quantile loss (pinball loss) over all quantile levels τ ∈ [0,1]. This connection bridges distributional forecasting and quantile regression.

• Integral representation: CRPS(F, y) = 2 ∫₀¹ QS_τ(F⁻¹(τ), y) dτ, where QS_τ is the quantile score at level τ
• Quantile forecast equivalence: Minimizing CRPS is equivalent to simultaneously minimizing the pinball loss at every quantile level
• Practical implication: A model that produces well-calibrated quantile forecasts at all levels will achieve a low CRPS

This relationship enables the use of quantile regression techniques—including quantile random forests and gradient boosting machines—as building blocks for CRPS-optimal forecasting systems in high-frequency trading applications.

CRPS and the logarithmic score (ignorance score) are both strictly proper, but they differ fundamentally in their sensitivity to tail events and distributional assumptions.

• Log score: Requires a full probability density function; assigns infinite penalty to zero-density events; highly sensitive to the tails
• CRPS: Requires only a CDF or samples; finite penalty for all outcomes; less sensitive to tail misspecification
• Selection guidance: Use log score when tail accuracy is paramount and a well-specified density is available; use CRPS when robustness to distributional misspecification is required or when evaluating ensemble forecasts

In high-frequency trading, where tick-level return distributions exhibit extreme kurtosis and models are often misspecified, CRPS provides a more stable and reliable evaluation signal than the log score.

A strictly proper scoring rule that evaluates a probabilistic forecast by taking the negative logarithm of the predicted probability density assigned to the observed outcome. Unlike CRPS, which compares cumulative distributions, the Log Score is a local scoring rule—it only evaluates the density at the observed value, making it highly sensitive to tail risk predictions.

• Formula: -log(p(y)) where p(y) is the predicted density at the observation
• Penalizes overconfidence severely when a low-probability event occurs
• Favored in information-theoretic frameworks and maximum likelihood estimation

The mean squared error for probabilistic forecasts of binary events. It measures the average squared difference between the forecast probability and the actual outcome (0 or 1). While CRPS generalizes this to continuous variables, the Brier Score remains the standard for evaluating probability forecasts of discrete events like default events or directional market moves.

• Range: 0 (perfect) to 1 (worst)
• Decomposable into reliability, resolution, and uncertainty components
• A Brier Score of 0.25 is equivalent to a coin flip

A diagnostic tool that transforms forecast-observation pairs into a uniform distribution if the forecast is calibrated. By applying the predicted CDF to the observed value, the resulting PIT values should follow U(0,1). Deviations from uniformity reveal specific miscalibration patterns.

• U-shaped histogram: Underdispersed forecasts (overconfident)
• Hump-shaped histogram: Overdispersed forecasts (underconfident)
• Asymmetric shape: Systematic bias in the forecast distribution
• Used to generate calibration diagnostics before computing CRPS

A scoring rule that evaluates a forecast for a specific quantile of the predictive distribution. The CRPS can be expressed as the integral of the Quantile Score over all quantile levels from 0 to 1. This decomposition makes CRPS particularly useful for Value-at-Risk (VaR) and Expected Shortfall backtesting.

• Formula: (y - q)(τ - 1{y < q}) where τ is the target quantile
• Asymmetric penalty: overprediction and underprediction penalized differently
• Foundation for quantile regression and non-parametric distributional forecasting

The practice of evaluating forecasts generated by an ensemble of deterministic models rather than a single predictive distribution. CRPS naturally handles ensemble forecasts by treating the empirical CDF of ensemble members as the predictive distribution. This is critical in meteorology and increasingly in multi-model trading ensembles.

• CRPS reduces to the mean absolute error when the ensemble has one member
• Fair CRPS variant corrects for finite ensemble size bias
• Enables direct comparison between ensemble systems and single probabilistic models

A scoring rule is strictly proper if the expected score is uniquely maximized (or minimized) when the forecaster reports their true belief distribution. CRPS is strictly proper, meaning it incentivizes honest reporting and cannot be gamed. This property is essential for forecast evaluation and model selection in adversarial settings like financial markets.

• Improper rules (like absolute error on point forecasts) encourage hedging
• Strict propriety ensures the optimal strategy is to predict the true distribution
• Forms the theoretical foundation for decision-theoretic model comparison