In Pursuit of Ideal Model Selection for High-Dimensional Linear Regression

Abstract: The fundamental importance of model specification has motivated researchers to study different aspects of this problem. One of which is the task of model selection from the set of available competing models. In this regard, several successful model selection criteria have been developed for the classical setting in which the number of measurements is much larger than the parameter space. However, when the number of measurements is comparable with the size of the dimension of the parameter space, these criteria are too liberal and prone to overfitting.In this thesis, we consider the problem of model selection for the high-dimensional setting in which the number of measurements is much smaller than the dimension of the parameter space. Inspired by previous work in this area, we propose a new model selection criterion based on the Fisher information. We analyze the performance of our criterion as the number of measurements increases to infinity as well as when the noise variance decreases to zero. We prove that the proposed criterion is consistent in selecting the true model in both scenarios. Besides, we conceive a computationally affordable algorithm to execute our model selection criterion. This algorithm utilizes the solution path of Lasso to narrow the set of all plausible combinatorial models down to a few ones. Interestingly, this algorithm also can be used for choosing the regularization parameter in the Lasso estimator properly. The empirical results support our theoretical findings. We also practice the task of model selection in situations where there are multiple measurement vectors available. Here, we also allow the elements of the noise vector to be spatially correlated. For such situations, we propose a non-negative Lasso estimator that is inspired by covariance matching techniques. Here, to tune the corresponding regularization parameter, we use our model selection criterion that has been introduced earlier. Empirical results show that our non-negative Lasso estimator can correctly select the true model when a relatively small number of measurement vectors are available. Moreover, the empirical results show that our proposed method is rather insensitive to a high correlation between the columns of the design matrix. In the last part of the thesis, we apply some of the theories and tools developed for model selection in the previous chapters to the problem of change point detection for noisy piecewise constant signals. In more details, we first consider the previously proposed change point estimation method, fused Lasso, and explain why it cannot guarantee the detection of the true change points. Then, we propose a normalized version of fused Lasso that is obtained by normalizing the columns of the sensing matrix of the Lasso equivalent. We analyze the performance of the proposed method, and in particular, we show that it is consistent in detecting change points as the noise variance tends to zero. Finally, we show numerical experiments that support our theoretical findings.