Search Results
Working Paper
The Anatomy of Out-of-Sample Forecasting Accuracy
We develop metrics based on Shapley values for interpreting time-series forecasting models, including“black-box” models from machine learning. Our metrics are model agnostic, so that they are applicable to any model (linear or nonlinear, parametric or nonparametric). Two of the metrics, iShapley-VI and oShapley-VI, measure the importance of individual predictors in fitted models for explaining the in-sample and out-of-sample predicted target values, respectively. The third metric is the performance-based Shapley value (PBSV), our main methodological contribution. PBSV measures the ...
Working Paper
The Anatomy of Out-of-Sample Forecasting Accuracy
We introduce the performance-based Shapley value (PBSV) to measure the contributions of individual predictors to the out-of-sample loss for time-series forecasting models. Our new metric allows a researcher to anatomize out-of-sample forecasting accuracy, thereby providing valuable information for interpreting time-series forecasting models. The PBSV is model agnostic—so it can be applied to any forecasting model, including "black box" models in machine learning, and it can be used for any loss function. We also develop the TS-Shapley-VI, a version of the conventional Shapley value that ...
Working Paper
Spatial Dependence and Data-Driven Networks of International Banks
This paper computes data-driven correlation networks based on the stock returns of international banks and conducts a comprehensive analysis of their topological properties. We first apply spatial-dependence methods to filter the effects of strong common factors and a thresholding procedure to select the significant bilateral correlations. The analysis of topological characteristics of the resulting correlation networks shows many common features that have been documented in the recent literature but were obtained with private information on banks? exposures. Our analysis validates these ...
Working Paper
Deep Neural Network Estimation in Panel Data Models
In this paper we study neural networks and their approximating power in panel data models. We provide asymptotic guarantees on deep feed-forward neural network estimation of the conditional mean, building on the work of Farrell et al. (2021), and explore latent patterns in the cross-section. We use the proposed estimators to forecast the progression of new COVID-19 cases across the G7 countries during the pandemic. We find significant forecasting gains over both linear panel and nonlinear time-series models. Containment or lockdown policies, as instigated at the national level by governments, ...
Working Paper
ChatMacro: Evaluating Inflation Forecasts of Generative AI
Recent research suggests that generic large language models (LLMs) can match the accuracy of traditional methods when forecasting macroeconomic variables in pseudo out-of-sample settings generated via prompts. This paper assesses the out-of-sample forecasting accuracy of LLMs by eliciting real-time forecasts of U.S. inflation from ChatGPT. We find that out-of-sample predictions are largely inaccurate and stale, even though forecasts generated in pseudo out-of-sample environments are comparable to existing benchmarks. Our results underscore the importance of out-of-sample benchmarking for LLM ...
Working Paper
Generative AI at the Crossroads: Light Bulb, Dynamo, or Microscope?
With the advent of generative AI (genAI), the potential scope of artificial intelligence has increased dramatically, but the future effect of genAI on productivity remains uncertain. The effect of the technology on the innovation process is a crucial open question. Some inventions, such as the light bulb, temporarily raise productivity growth as adoption spreads, but the effect fades when the market is saturated; that is, the level of output per hour is permanently higher but the growth rate is not. In contrast, two types of technologies stand out as having longer-lived effects on ...
Working Paper
LLM on a Budget: Active Knowledge Distillation for Efficient Classification of Large Text Corpora
Large Language Models (LLMs) are highly accurate in classification tasks, however, substantial computational and financial costs hinder their large-scale deployment in dynamic environments. Knowledge Distillation (KD) where a LLM ""teacher"" trains a smaller and more efficient ""student"" model, offers a promising solution to this problem. However, the distillation process itself often remains costly for large datasets, since it requires the teacher to label a vast number of samples while incurring significant token consumption. To alleviate this challenge, in this work we explore the ...
Working Paper
Validating Large Language Model Annotations
This paper proposes a validation framework for LLM-generated measurements when reliable benchmarks are unavailable. Validity is established by testing whether an LLM can reconstruct passages from annotated labels while maintaining semantic consistency with the original text. The framework avoids circular reasoning by establishing testable prerequisite properties that must be met for a validation to be considered successful. Application to news article data demonstrates that the framework serves as a practical alternative to human benchmarking, which offers advantages in objectivity, ...
Working Paper
Machine Learning, the Treasury Yield Curve and Recession Forecasting
We use machine learning methods to examine the power of Treasury term spreads and other financial market and macroeconomic variables to forecast US recessions, vis-à-vis probit regression. In particular we propose a novel strategy for conducting cross-validation on classifiers trained with macro/financial panel data of low frequency and compare the results to those obtained from standard k-folds cross-validation. Consistent with the existing literature we find that, in the time series setting, forecast accuracy estimates derived from k-folds are biased optimistically, and cross-validation ...
Working Paper
Integrating Prediction and Attribution to Classify News
Recent modeling developments have created tradeoffs between attribution-based models, models that rely on causal relationships, and “pure prediction models†such as neural networks. While forecasters have historically favored one technology or the other based on comfort or loyalty to a particular paradigm, in domains with many observations and predictors such as textual analysis, the tradeoffs between attribution and prediction have become too large to ignore. We document these tradeoffs in the context of relabeling 27 million Thomson Reuters news articles published between 1996 ...