Classical Probability of Overfitting with Information Criteria: Estimations with Chilean Macroeconomic Series

Carlos A. Medel

doi:10.4067/S0718-88702015000100004

Authors

Carlos A. Medel Universidad de Nottingham

DOI:

https://doi.org/10.4067/S0718-88702015000100004

Keywords:

Nonparametric modelling, information criteria, overfitting, out-of-sample analysis

Abstract

This paper provides, via Monte Carlo simulations, estimates of the classical probability of overfitting under an autoregressive environment (AR), using the information criteria (IC) of Akaike, Schwarz and Hannan-Quinn (AIC, BIC and HQ), calibrated with Chilean data of total inflation, core inflation, IMACEC, and monthly return of the nominal exchange rate Chilean peso- American dollar. This probability corresponds to the number of times when a candidate model has a strictly greater number of coefficients than the true model, divided by the total number of searches. The results indicate that the increased risk of overfitting is obtained with the AIC, followed by HQ and finally the BIC. The highest probability of overfitting is achieved with the AIC, reaching 32 and 30% with the exchange rate and IMACEC, respectively, followed by 25 and 22% for total and core inflation. Considering the three IC, it is more likely to obtain an overfitted model by just one coefficient. Also, it is more likely that the overfitting does not exceed 10 coefficients. These results are important as quantifying the extent to which these variables are subject to overfitting risk when represented by AR models. The potential problems carried by overfitting includes: (i) the spurious regression, (ii) distor tthe estimation of impulse response function, and (iii) affect the predictive accuracy of the variable of interest. The latter problem is analyzed in detail.

Downloads

Download data is not yet available.

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/tac.1974.1100705

Box, G. E. P., & Jenkins, G. M. (1970). Time series analysis: Forecasting and control. Holden-Day.

Canova, F. (2007). Methods for applied macroeconomic research. Princeton University Press.

Cavanaugh, J. E. (1997). Unifying the derivations for the Akaike and corrected Akaike information criteria. Statistics and Probability Letters, 33(2), 201–208. https://doi.org/10.1016/s0167-7152(96)00128-9

Clark, T. E. (2004). Can out-of-sample forecast comparisons help prevent overfitting? Journal of Forecasting, 23(2), 115–139.

Elliott, G., & Timmermann, A. (2008). Economic forecasting. Journal of Economic Literature, 46(1), 3–56. https://doi.org/10.1257/jel.46.1.3

Faust, J., & Wright, J. (2013). Forecasting inflation. In G. Elliott & A. Timmermann (Eds.), Handbook of economic forecasting (Vol. 2, pp. 2–56). Elsevier. https://doi.org/10.1016/b978-0-444-53683-9.00001-3

Granger, C. W. J., & Jeon, Y. (2004). Forecasting performance of information criteria with many macro series. Journal of Applied Statistics, 31(10), 1227–1240. https://doi.org/10.1080/0266476042000285495

Granger, C. W. J., & Jeon, Y. (2006). Dynamics of model overfitting measured in terms of autoregressive roots. Journal of Time Series Analysis, 27(3), 347–365. https://doi.org/10.1111/j.1467-9892.2006.00468.x

Granger, C. W. J., & Newbold, P. (1974). Spurious regressions in econometrics. Journal of Econometrics, 2(1), 111–120. https://doi.org/10.1016/0304-4076(74)90034-7

Hannan, E. J., & Quinn, B. G. (1979). The determination of the order of an autoregression. Journal of the Royal Statistical Society, Series B, 41, 190–195.

Hansen, P. R. (2009). In-sample fit and out-of-sample fit: Their joint distribution and its implications for model selection [Unpublished manuscript]. Department of Economics, Stanford University.

Hawkins, D. M. (2004). The problem of overfitting. Journal of Chemical Information and Computer Sciences, 44(1), 1–12. https://doi.org/10.1021/ci0342472

Hoover, K. D., & Perez, S. J. (1999). Data mining reconsidered: Encompassing and the general-to-specific approach to specification search. The Econometrics Journal, 2(2), 167–191. https://doi.org/10.1111/1368-423x.00025

Hurvich, C. M., & Tsai, C.-L. (1993). A corrected Akaike information criterion for vector autoregressive model selection. Journal of Time Series Analysis, 14(3), 271–279. https://doi.org/10.1111/j.1467-9892.1993.tb00144.x

Kilian, L. (2001). Impulse response analysis in vector autoregressions with unknown lag order. Journal of Forecasting, 20(3), 161–179. https://doi.org/10.1002/1099-131x(200104)20:3<161::aid-for770>3.0.co;2-x

Koehler, A. B., & Murphree, E. S. (1988). A comparison of the Akaike and Schwarz criteria for selecting model order. Journal of the Royal Statistical Society, Series C (Applied Statistics), 37(2), 187–195.

Kullback, S. (1968). Information theory and statistics. Dover.

Kullback, S., & Leibler, R. A. (1951). On information and sufficiency. Annals of Mathematical Statistics, 22(1), 79–86. https://doi.org/10.1214/aoms/1177729694

Lütkepohl, H. (1985). Comparison of criteria for estimating the order of a vector autoregressive process. Journal of Time Series Analysis, 6(1), 35–52. https://doi.org/10.1111/j.1467-9892.1985.tb00396.x

McQuarrie, A. D. (1999). A small-sample correction for the Schwarz SIC model selection criterion. Statistics and Probability Letters, 44(1), 79–86. https://doi.org/10.1016/s0167-7152(98)00294-6

Medel, C. A. (2011). ¿Akaike o Schwarz? ¿Cuál utilizar para predecir el PIB chileno? [Akaike or Schwarz? Which one to use to forecast Chilean GDP?]. Monetaria, XXXIV(4), 591–615.

Medel, C. A. (2013). How informative are in-sample information criteria to forecasting? The case of Chilean GDP. Latin American Journal of Economics, 50(1), 133–161. https://doi.org/10.7764/laje.50.1.133

Medel, C. A., & Salgado, S. (2013). Does the BIC estimate and forecast better than the AIC? Economic Analysis Review, 28(1), 47–64.

Nishii, R. (1984). Asymptotic properties of criteria for selection of variables in multiple regression. Annals of Statistics, 12(2), 758–765. https://doi.org/10.1214/aos/1176346522

Schwarz, G. E. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461–464. https://doi.org/10.1214/aos/1176344136

Seghouane, A. K. (2006). A note on overfitting properties of KIC and KICc. Signal Processing, 86(10), 3055–3060. https://doi.org/10.1016/j.sigpro.2006.01.002

Shibata, R. (1976). Selection of the order of an autoregressive model by Akaike information criterion. Biometrika, 63(1), 117–126. https://doi.org/10.2307/2335091

Xu, C. J., & McLeod, I. (2012). Further asymptotic properties of the generalized information criteria. Electronic Journal of Statistics, 6, 656–663.