Pitfalls of Post-model-selection Testing: Experimental Quantification

Abstract

Traditional specification testing does not always improve subsequent inference. We demonstrate by means of computer experiments under which circumstances, and how severely, data-driven model selection can destroy the size properties of subsequent parameter tests, if they are used without adjusting for the model-selection step. The investigated models are representative of macroeconometric and microeconometric workhorses. The model selection procedures include information criteria as well as sequences of significance tests ("general-to-specific"). We find that size distortions can be particularly large when competing models are close, with closeness being defined relatively to the sample size.