Search Results
Working Paper
The Anatomy of Out-of-Sample Forecasting Accuracy
We develop metrics based on Shapley values for interpreting time-series forecasting models, including“black-box” models from machine learning. Our metrics are model agnostic, so that they are applicable to any model (linear or nonlinear, parametric or nonparametric). Two of the metrics, iShapley-VI and oShapley-VI, measure the importance of individual predictors in fitted models for explaining the in-sample and out-of-sample predicted target values, respectively. The third metric is the performance-based Shapley value (PBSV), our main methodological contribution. PBSV measures the ...
Working Paper
Machine Learning, the Treasury Yield Curve and Recession Forecasting
We use machine learning methods to examine the power of Treasury term spreads and other financial market and macroeconomic variables to forecast US recessions, vis-à-vis probit regression. In particular we propose a novel strategy for conducting cross-validation on classifiers trained with macro/financial panel data of low frequency and compare the results to those obtained from standard k-folds cross-validation. Consistent with the existing literature we find that, in the time series setting, forecast accuracy estimates derived from k-folds are biased optimistically, and cross-validation ...
Working Paper
Evaluating Local Language Models: An Application to Bank Earnings Calls
This study evaluates the performance of local large language models (LLMs) in interpreting financial texts, compared with closed-source, cloud-based models. We first introduce new benchmarking tasks for assessing LLM performance in analyzing financial and economic texts and explore the refinements needed to improve its performance. Our benchmarking results suggest local LLMs are a viable tool for general natural language processing analysis of these texts. We then leverage local LLMs to analyze the tone and substance of bank earnings calls in the post-pandemic era, including calls conducted ...
Working Paper
Integrating Prediction and Attribution to Classify News
Recent modeling developments have created tradeoffs between attribution-based models, models that rely on causal relationships, and “pure prediction models†such as neural networks. While forecasters have historically favored one technology or the other based on comfort or loyalty to a particular paradigm, in domains with many observations and predictors such as textual analysis, the tradeoffs between attribution and prediction have become too large to ignore. We document these tradeoffs in the context of relabeling 27 million Thomson Reuters news articles published between 1996 ...
Working Paper
Macroeconomic Indicator Forecasting with Deep Neural Networks
Economic policymaking relies upon accurate forecasts of economic conditions. Current methods for unconditional forecasting are dominated by inherently linear models {{p}} that exhibit model dependence and have high data demands. {{p}} We explore deep neural networks as an {{p}} opportunity to improve upon forecast accuracy with limited data and while remaining agnostic as to {{p}} functional form. We focus on predicting civilian unemployment using models based on four different neural network architectures. Each of these models outperforms bench- mark models at short time horizons. One model, ...
Working Paper
Deep Neural Network Estimation in Panel Data Models
In this paper we study neural networks and their approximating power in panel data models. We provide asymptotic guarantees on deep feed-forward neural network estimation of the conditional mean, building on the work of Farrell et al. (2021), and explore latent patterns in the cross-section. We use the proposed estimators to forecast the progression of new COVID-19 cases across the G7 countries during the pandemic. We find significant forecasting gains over both linear panel and nonlinear time-series models. Containment or lockdown policies, as instigated at the national level by governments, ...