We investigate the factor structure of the term structure of interest rates and argue that characterizing the minimal dimension of the data generating process is more challenging than currently appreciated. As a result, inference procedures for yield curve models that commit to a parsimoniously parameterized factor structure may be omitting important information about the underlying true factor space. To circumvent these difficulties, we introduce a novel nonparametric bootstrap that is robust to general forms of time and cross-sectional dependence and conditional heteroskedasticity of unknown form. We show that our bootstrap procedure is asymptotically valid and exhibits excellent finite-sample properties in simulations. We demonstrate the applicability of our results in two empirical exercises: first, we show that measures of equity market tail risk and the state of the macroeconomy predict bond returns beyond the level or slope of the yield curve; second, we provide a bootstrap-based bias correction and confidence intervals for the probability of recession based on the shape of the yield curve. Our results apply more generally to all assets with a finite maturity structure.