Bootstrapping LASSO estimators under variable selection consistency in high dimensions and some higher order refinements

Abstract

We consider statistical inference based on the LASSO (cf. Tibshirani (J. Roy. Statist. Soc. Ser. B, Methodol. 58 (1996) 267–288)) in high dimensional regression problems. It is well known that the LASSO produces biased estimator of the regression parameter. The bias problem is further exacerbated when the LASSO has the variable selection consistency property (cf. Lahiri (Ann. Statist. 49 (2021) 820–844)). The centered and scaled point estimators are dominated by the bias term and fail to converge to any non-degenerate limit distribution. In contrast, we show that the LASSO interval estimators based on the Bootstrap are surprisingly accurate. Here, we consider two variants of the Bootstrap, namely, the Residual Bootstrap and the Perturbation Bootstrap, and show that under some regularity conditions, the Bootstrap approximations generated by both variants automatically adjust for the effects of the bias and are second order correct. This is an important finding as centered and scaled LASSO estimators under variable selection consistency fail to have a non-degenerate limit distribution and the Bootstrap approximations provide a viable way of constructing valid confidence intervals. Building on these second order results, we next consider construction of two-sided symmetric Bootstrap confidence intervals and show that with suitable choices of the studentized pivotal quantities, Bootstrap based two-sided symmetric confidence intervals attain a level of accuracy O(n−2) where n is the sample size. Thus, even with penalization and with diverging model parameter dimension, the performance of the Bootstrap here remains comparable to its finite dimensional counterpart in the traditional Smooth Function model (cf. Hall (The Bootstrap and Edgeworth Expansion (1992) Springer-Verlag)). We also establish similar results for the heteroscedastic case for the Perturbation Bootstrap and report results from a moderate simulation study in support of the theoretical findings.