The Anatomy of Out-of-Sample Forecasting Accuracy

RePEc Short ID: pbo994
ORCID iD: 0000-0002-6560-1830

Borup, Daniel; Goulet Coulombe, Philippe; Montes Schütte, Erik Christian; Rapach, David E.; Schwenk-Nebbe, Sander

doi:10.29338/wp2022-16b

Working Paper

The Anatomy of Out-of-Sample Forecasting Accuracy

Abstract: We introduce the performance-based Shapley value (PBSV) to measure the contributions of individual predictors to the out-of-sample loss for time-series forecasting models. Our new metric allows a researcher to anatomize out-of-sample forecasting accuracy, thereby providing valuable information for interpreting time-series forecasting models. The PBSV is model agnostic—so it can be applied to any forecasting model, including "black box" models in machine learning, and it can be used for any loss function. We also develop the TS-Shapley-VI, a version of the conventional Shapley value that gauges the importance of predictors for explaining the in-sample predictions in the entire sequence of fitted models that generates the time series of out-of-sample forecasts. We then propose the model accordance score to compare predictor ranks based on the TS-Shapley-VI and PBSV, thereby linking the predictors' in-sample importance to their contributions to out-of-sample forecasting accuracy. We illustrate our metrics in an application forecasting US inflation.

Keywords: model interpretation; Shapley value; predictor importance; loss function; machine learning; inflation;

JEL Classification: C22; C45; C52; C53; E31; E37;

https://doi.org/10.29338/wp2022-16b

Status: Published in 2024