Backtesting

Unlike standard k-fold cross-validation, backtesting respects the temporal ordering of data to prevent look-ahead bias. The model is trained on a historical window and tested on a subsequent, unseen period. Common strategies include:

• Expanding Window: The training set grows to include all prior data, mimicking a production model that accumulates history over time.
• Rolling Window: A fixed-size training window slides forward, discarding old data to adapt to recent patterns.
• Walk-Forward Optimization: A specific form of rolling window where the model is retrained at each step, rigorously testing stability.

Standard accuracy measures quantify the deviation between the point forecast and the actual observed value. Key metrics include:

• MAE (Mean Absolute Error): The average absolute difference, providing an intuitive measure in the original units.
• RMSE (Root Mean Squared Error): Penalizes large errors more heavily than MAE, making it sensitive to outliers.
• MAPE (Mean Absolute Percentage Error): Expresses error as a percentage, useful for communicating with business stakeholders but undefined when actuals are zero.
• sMAPE (Symmetric MAPE): Addresses MAPE's asymmetry by dividing by the average of forecast and actual.

For models that output a full prediction distribution, point metrics are insufficient. Proper scoring rules evaluate the entire distribution:

• Continuous Ranked Probability Score (CRPS): Measures the integrated squared difference between the predicted CDF and the empirical observation. It generalizes MAE for probabilistic forecasts.
• Pinball Loss: Evaluates a specific quantile forecast. It asymmetrically penalizes over-prediction and under-prediction based on the target quantile, directly optimizing for service-level goals.
• Prediction Interval Coverage: Checks if the true value falls within the predicted interval at the nominal rate (e.g., a 90% interval should contain the truth 90% of the time).

A model with low aggregate error can still be systematically flawed. Residual analysis examines the pattern of errors over time:

• Mean Error (ME): A non-zero value indicates persistent forecast bias (consistent over-forecasting or under-forecasting).
• Autocorrelation of Residuals: Significant autocorrelation suggests the model is not capturing all temporal structure, leaving exploitable signal in the errors.
• Heteroscedasticity: A funnel shape in residual plots indicates that error variance changes with the forecast level, violating assumptions of many statistical models.

When models use external features like pricing, promotions, or weather, backtesting must ensure causal integrity. Using future information for a past prediction constitutes data leakage. The process must replicate the production state:

• Feature Lagging: Ensure covariates are shifted to reflect the information available at the forecast origin.
• Ex-Ante vs. Ex-Post: Distinguish between forecasts made with actual known future covariates (ex-post) for diagnostic purposes and those made with forecasted covariates (ex-ante) for realistic performance assessment.
• Scenario Testing: Apply historical shock events (e.g., a supplier bankruptcy) to the feature set to evaluate model resilience under stress.

Statistical accuracy does not guarantee business value. The ultimate backtest translates forecast errors into operational and financial impact:

• Inventory Simulation: Feed backtested forecasts into an inventory policy (e.g., newsvendor model) to simulate realized holding costs, stockout events, and service levels.
• Waste vs. Shortage Cost Curves: For perishable goods, evaluate the trade-off between spoilage from over-forecasting and lost revenue from under-forecasting.
• Decision-Focused Evaluation: Weight errors by their asymmetric business cost. A stockout at a high-margin customer is penalized more heavily than an overstock of a low-cost item.

The gold-standard backtesting methodology that simulates real-world deployment by incrementally rolling the training window forward. At each step, the model is trained on data up to time t, predicts for t+1, then the window expands to include t+1 before retraining. This prevents look-ahead bias and provides a realistic measure of how a model adapts to concept drift. Unlike standard k-fold cross-validation, walk-forward preserves the temporal order of observations, making it essential for non-stationary supply chain data.

A strictly proper scoring rule that evaluates the full predictive distribution, not just a point estimate. CRPS measures the integrated squared difference between the predicted cumulative distribution function and a step function at the observed value. Lower scores indicate better probabilistic calibration. Unlike Mean Absolute Error, CRPS rewards models that correctly express epistemic uncertainty—penalizing both overconfident wrong predictions and underconfident vague ones.

An asymmetric loss function central to quantile regression backtesting. It applies a weight of τ to positive errors and (1-τ) to negative errors, where τ is the target quantile. For a 95th percentile forecast, under-prediction is penalized 19 times more heavily than over-prediction. This allows backtesters to verify that prediction intervals achieve their nominal coverage—a 90% interval should contain the actual value exactly 90% of the time across a sufficient test window.

The post-hoc adjustment process that ensures hierarchical coherence across backtested forecasts. When bottom-up SKU-level predictions are aggregated, they must mathematically equal top-down category forecasts. Reconciliation algorithms like MinT (Minimum Trace) optimally distribute errors across the hierarchy using the covariance structure of base forecast errors. Without reconciliation, backtesting results at different aggregation levels will show inconsistent accuracy metrics, undermining trust in the system.

A statistical hypothesis test that determines whether the difference in forecast accuracy between two competing models is significant or merely due to random chance. The test accounts for autocorrelation in forecast errors and can be applied to any loss function. In backtesting, it answers the critical question: Is Model A genuinely superior to Model B, or did it just get lucky on this particular historical window?

The most insidious failure mode in backtesting, where information that would not have been available at prediction time leaks into the training process. Common causes include: - Scaling features using future min/max values - Applying PCA fit on the entire dataset - Using supplier lead times that were revised retroactively - Training on demand data that was later adjusted for returns Rigorous temporal point-in-time snapshots of the feature store are the primary defense.