The Anatomy of Out-of-Sample Forecasting Accuracy

RePEc Short ID: pbo994
ORCID iD: 0000-0002-6560-1830

Borup, Daniel; Goulet Coulombe, Philippe; Montes Schütte, Erik Christian; Rapach, David E.; Schwenk-Nebbe, Sander

doi:10.29338/wp2022-16

Working Paper

The Anatomy of Out-of-Sample Forecasting Accuracy

Abstract: We develop metrics based on Shapley values for interpreting time-series forecasting models, including “black-box” models from machine learning. Our metrics are model agnostic, so that they are applicable to any model (linear or nonlinear, parametric or nonparametric). Two of the metrics, iShapley-VI and oShapley-VI, measure the importance of individual predictors in fitted models for explaining the in-sample and out-of-sample predicted target values, respectively. The third metric is the performance-based Shapley value (PBSV), our main methodological contribution. PBSV measures the contributions of individual predictors in fitted models to the out-of-sample loss and thereby anatomizes out-of-sample forecasting accuracy. In an empirical application forecasting US inflation, we find important discrepancies between individual predictor relevance according to the in-sample iShapley-VI and out-of-sample PBSV. We use simulations to analyze potential sources of the discrepancies, including overfitting, structural breaks, and evolving predictor volatilities.

Keywords: variable importance; out-of-sample performance; Shapley value; loss function; machine learning; inflation;

JEL Classification: C22; C45; C53; E37; G17;

https://doi.org/10.29338/wp2022-16

Status: Published in 2022