Does the Bic Estimate and Forecast Better than the Aic?*

Carlos A. Medel; Sergio C. Salgado

doi:10.4067/S0718-88702013000100003

Authors

Carlos A. Medel Central Bank of Chile
Sergio C. Salgado University of Minnesota

DOI:

https://doi.org/10.4067/S0718-88702013000100003

Keywords:

AIC, BIC, information criteria, time-series models, overfitting, forecast comparison, joint hypothesis testing

Abstract

We test two questions: (i) Is the Bayesian Information Criterion (BIC) more parsimonious than Akaike Information Criterion (AIC)? and (ii) Is BIC better than AIC for forecasting purposes? By using simulated data, we provide statistical inference of both hypotheses individually and then jointly with a multiple hypotheses testing procedure to control better for type-I error. Both testing procedures deliver the same result: The BIC shows an in- and out-of-sample superiority over AIC only in a long-sample context.

Downloads

Download data is not yet available.

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/tac.1974.1100705

Bhansali, R. J., & Downham, D. Y. (1977). Some properties of the order of an autoregressive model selected by a generalization of Akaike's EPF criterion. Biometrika, 64(3), 547–551. https://doi.org/10.1093/biomet/64.3.547

Bickel, P., & Zhang, P. (1992). Variable selection in nonparametric regression with categorical covariates. Journal of the American Statistical Association, 87(417), 90–97. https://doi.org/10.2307/2290456

Bozdogan, H. (1987). Model selection and Akaike's information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52(3), 345–370. https://doi.org/10.1007/bf02294361

Burnham, K. P., & Anderson, D. R. (1998). Model selection and inference: A practical information theoretic approach. Springer.

Canova, F. (2007). Methods for applied macroeconomic research. Princeton University Press.

Chen, L., Giannakouros, P., & Yang, Y. (2007). Model combining in factorial data analysis. Journal of Statistical Planning and Inference, 137(9), 2920–2934. https://doi.org/10.1016/j.jspi.2006.10.005

Clark, T. E. (2004). Can out-of-sample forecast comparisons help to prevent overfitting? Journal of Forecasting, 23(2), 115–139.

Clark, T. E., & McCracken, M. (2001). Tests of equal forecast accuracy and encompassing for nested models. Journal of Econometrics, 105(1), 85–110. https://doi.org/10.1016/s0304-4076(01)00071-9

Corradi, V., & Distaso, W. (2011). Multiple forecast model evaluation. In M. P. Clements & D. F. Hendry (Eds.), The Oxford handbook of economic forecasting (pp. 391–414). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780195398649.013.0014

Diebold, F. X., & Mariano, R. (1995). Comparing predictive accuracy. Journal of Business and Economic Statistics, 13(3), 253–263.

Fischer, S. (1981). Relative shocks, relative price volatility, and inflation. Brookings Papers on Economic Activity, 2, 381–431.

Geweke, J., & Meese, R. (1981). Estimating regression models of finite but unknown order. International Economic Review, 22(1), 55–70. https://doi.org/10.2307/2526135

Gordon, R. J., & King, S. R. (1982). The output cost of disinflation in traditional and vector autoregressive models. Brookings Papers on Economic Activity, 13(1), 205–244. https://doi.org/10.2307/2534320

Granger, C. W. J., & Jeon, Y. (2004). Forecasting performance of information criteria with many macro series. Journal of Applied Statistics, 31(10), 1227–1240. https://doi.org/10.1080/0266476042000285495

Hansen, P. R. (2005). A test of superior predictive ability. Journal of Business and Economic Statistics, 23, 365–380.

Hansen, P. R. (2009). In-sample fit and out-of-sample fit: Their joint distribution and its implications for model selection [Unpublished manuscript]. Stanford University.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.

Hurvich, C. M., & Tsai, C.-L. (1993). A corrected Akaike information criterion for vector autoregressive model selection. Journal of Time Series Analysis, 14(3), 271–279. https://doi.org/10.1111/j.1467-9892.1993.tb00144.x

Karagrigoriou, A., Mattheou, K., & Vonta, I. (2011). On asymptotic properties of AIC variants with applications. American Open Journal of Statistics, 1(02), 105–109. https://doi.org/10.4236/ojs.2011.12012

Kilian, L. (2001). Impulse response analysis in vector autoregressions with unknown lag order. Journal of Forecasting, 20(3), 161–179. https://doi.org/10.1002/1099-131x(200104)20:3<161::aid-for770>3.0.co;2-x

Koehler, A. B., & Murphree, E. S. (1988). A comparison of the Akaike and Schwarz criteria for selecting model order. Journal of the Royal Statistical Society, Series C (Applied Statistics), 37(2), 187–195.

Kuha, J. (2004). AIC and BIC: Comparison of assumptions and performance. Sociological Methods and Research, 33(2), 188–229. https://doi.org/10.1177/0049124103262065

Lütkepohl, H. (1985). Comparison of criteria for estimating the order of a vector autoregressive process. Journal of Time Series Analysis, 6(1), 35–52. https://doi.org/10.1111/j.1467-9892.1985.tb00396.x

Markon, K. E., & Krueger, R. F. (2004). An empirical comparison of information-theoretic selection criteria for multivariate behavior genetic models. Behavior Genetics, 34(6), 593–609. https://doi.org/10.1007/s10519-004-5587-0

Newey, W., & West, K. (1987). A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica, 55(3), 703–708. https://doi.org/10.2307/1913610

Nickelsburg, G. (1985). Small-sample properties of dimensionality statistics for fitting VAR models to aggregate economic data: A Monte Carlo study. Journal of Econometrics, 28(2), 183–192. https://doi.org/10.1016/0304-4076(85)90118-6

Nishii, R. (1984). Asymptotic properties of criteria for selection of variables in multiple regression. Annals of Statistics, 12(2), 758–765. https://doi.org/10.1214/aos/1176346522

Pincheira, P. (2011a). A joint test of superior predictive ability for Chilean inflation (Working Paper 620) [In Spanish]. Central Bank of Chile.

Pincheira, P. (2011b). A bunch of models, a bunch of nulls and inference about predictive ability (Working Paper 607). Central Bank of Chile.

Pincheira, P. (2012). Un test conjunto de superioridad predictiva para los pronósticos de inflación chilena [A joint test of superior predictive ability for Chilean inflation forecasts]. The Chilean Economy, 15(3), 4–39.

Pötscher, B. M., & Srinivasan, S. (1991). A comparison of order estimation procedures for ARMA models. Statistica Sinica, 4, 29–50.

Raffalovich, L. E., Deane, G. D., Armstrong, D., & Tsao, H.-S. (2008). Model selection procedures in social research: Monte-Carlo simulation results. Journal of Applied Statistics, 35(10), 1094–1114. https://doi.org/10.1080/03081070802203959

Rissanen, J. (1978). Modeling by shortest data description. Automatica, 14(5), 465–471. https://doi.org/10.1016/0005-1098(78)90005-5

Sargent, T., & Sims, C. (1977). Business cycle modeling without pretending to have too much a priori economic theory (Working Paper 55). Federal Reserve Bank of Minneapolis.

Sawa, T. (1978). Information criteria for discriminating among alternative regression models. Econometrica, 46(6), 1273–1282. https://doi.org/10.2307/1913828

Schwarz, G. E. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461–464. https://doi.org/10.1214/aos/1176344136

Shibata, R. (1976). Selection of the order of an autoregressive model by Akaike information criterion. Biometrika, 63(1), 117–126. https://doi.org/10.2307/2335091

Shibata, R. (1980). Asymptotically efficient selection of the order of the model for estimating parameters of a linear process. Annals of Statistics, 8(1), 147–164. https://doi.org/10.1214/aos/1176344897

Shibata, R. (1981). An optimal selection of regression variables. Biometrika, 68(1), 45–54. https://doi.org/10.1093/biomet/68.1.45

Shittu, O. I., & Asemota, M. J. (2009). Comparison of criteria for estimating the order of autoregressive process: A Monte Carlo approach. European Journal of Scientific Research, 30(3), 409–416.

Sims, C. (1980). Macroeconomics and reality. Econometrica, 48(1), 1–48. https://doi.org/10.2307/1912017

Stone, M. (1979). Comments on model selection criteria of Akaike and Schwarz. Journal of the Royal Statistical Society, Series B (Methodological), 41(2), 276–278.

Sullivan, R., Timmermann, A., & White, H. (1999). Data-snooping, technical trading rule performance, and the bootstrap. Journal of Finance, 54(5), 1647–1691. https://doi.org/10.1111/0022-1082.00163

Wasserman, L. (2000). Bayesian model selection and model averaging. Journal of Mathematical Psychology, 44(1), 92–107. https://doi.org/10.1006/jmps.1999.1278

Weakliem, L. D. (2004). Introduction to the special issue on model selection. Sociological Methods and Research, 33(2), 167–186. https://doi.org/10.1177/0049124104268642

West, K. (1996). Asymptotic inference about predictive ability. Econometrica, 64(5), 1067–1084.

White, H. (2000). A reality check for data snooping. Econometrica, 68(5), 1097–1126. https://doi.org/10.1111/1468-0262.00152

Wolak, F. A. (1987). An exact test for multiple inequality and equality constraints in the linear regression model. Journal of the American Statistical Association, 82(399), 782–793. https://doi.org/10.1080/01621459.1987.10478499

Wolak, F. A. (1989). Testing inequality constraints in linear econometric models. Journal of Econometrics, 31(2), 205–235. https://doi.org/10.1016/0304-4076(89)90094-8

Xu, C. J., & McLeod, I. (2012). Further asymptotic properties of the generalized information criteria. Electronic Journal of Statistics, 6, 656–663.

Yang, Y. (2003). Regression with multiple candidate models: Selecting or mixing? Statistica Sinica, 13, 783–809.

Yi, G., & Judge, G. (1988). Statistical model selection criteria. Economics Letters, 28(1), 47–51. https://doi.org/10.1016/0165-1765(88)90070-5

Zucchini, W. (2000). An introduction to model selection. Journal of Mathematical Psychology, 44(1), 41–46. https://doi.org/10.1006/jmps.1999.1276