A split-and-conquer variable selection approach for high-dimensional general semiparametric models with massive data

Abstract

Estimation and variable selection in partially linear models for massive data has been discussed by several authors. However, there does not seem to exist an established procedure for other semiparametric models, such as the semiparametric varying-coefficient linear model, the single index regression model, the partially linear errors-in-variables model, etc. In this paper, we propose a general procedure for variable selection in high-dimensional general semiparametric models by penalized semiparametric estimating equations. Under some regularity conditions, the oracle property is established, which the number of parameters is allowed to diverge. Furthermore, we also propose a split-and-conquer variable selection procedure for high-dimensional general semiparametric models with massive data. Under some weak regularity conditions, we establish the oracle property of the proposed procedure when the number of subsets does not grow too fast. What is more, the split-and-conquer procedure enjoys the oracle property as the penalized estimator by using all the dataset, and can substantially reduce computing time and computer memory requirements. The performance of the proposed method is illustrated via a real data application and numerical simulations.

Introduction

In the past two decades, revolutions in information technologies have produced many kinds of massive data. It is very challenging to extract useful features from a massive dataset since many statistics are difficult to compute by traditional algorithms or statistical packages when the massive dataset is too large to be stored in primary memory. In recent years, there have been active researches on developing methods to analyze massive data. Lin and Xi [15] developed an aggregated algorithm for estimating equation estimations in massive datasets using a divide-and-conquer strategy; Chen and Xie [2] considered a divide-and-conquer approach for generalized linear models where both the sample size $n$ and the number of covariates $p$ are large; Song and Liang [18] proposed a split-and-merge Bayesian variable selection method for ultrahigh dimensional linear regression models; Kleiner et al. [1] proposed the bag of little bootstraps approach for massive data; Liang et al. [14] developed a resampling-based method for the big geostatistical data; Schifano et al. [17] presented an online updating strategy for big data arising from online analytical processing; etc.

Variable selection is an interesting problem in statistical analysis. Variable selection for high dimensional regression is often treated with penalized likelihood procedures. The popular penalized likelihood procedures include the LASSO (Tibshirani [20]), the smoothly clipped absolute deviation (SCAD) (Fan and Li [4]), among others. Variable selection procedures for high-dimensional setting, where the dimension $p$ of the observations increases with the sample size $n$, are given by Fan and Peng [5], Tang and Leng [19], etc. The penalized procedure has been extended to generalized estimating equations through the penalized generalized estimating equations. Fu [8] proposed a generalization of the bridge penalty to generalized estimating equations. Johnson et al. [10] proposed a procedure for variable selection in semiparametric regression models by penalizing appropriate estimating functions. Wang [24] proposed the SCAD-penalized generalized estimating equations procedure in which the number of parameters is allowed to diverge.

In this paper, motivated by the variable selection procedure for estimating equations in Johnson et al. [10], we firstly extend the penalized procedure to general semiparametric models based on penalized semiparametric estimating functions with the case of the data dimension $p$ diverging. Johnson et al. [10] proposed a variable selection procedure for estimating equations. However, the combining estimating equation and the penalty function method by Johnson et al. [10] could not be used directly to semi-parametric estimating equations. For example, we consider the partially linear model $Y=X^{\top }\theta +f\left(T\right)+\varepsilon$, the semi-parametric estimating functions can be taken by $g(\mathbf{X},H\left(T\right),\theta )=\{X-E\left(X\right|T)\}[\{Y-E\left(Y\right|T)\}-\{X-E{\left(X\right|T)}^{\top }\theta \}]$. Since the corresponding semi-parametric estimating equations involve the unknown function $E\left(X\right|T)$ and $E\left(Y\right|T)$, thus we need to estimate the function $E\left(X\right|T)$ and $E\left(X\right|T)$ in advance. We propose a general procedure for variable selection in high-dimensional semiparametric models by penalized semiparametric estimating equations, including partially linear models, semiparametric varying-coefficient linear models, partially linear errors-in-variables models, etc.

Semiparametric models are a useful compromise between parametric and nonparametric models to mitigate the curse of dimensionality but still allow reasonable flexibility to specify functional form; see, e.g., Engle et al. [3]. For massive data setting, Zhao, Cheng and Liu [25] considered a partially linear framework for modelling massive heterogeneous data, and proposed to extract the common feature across all sub-populations while exploring heterogeneity of each sub-population. Lian, Zhao and Lv [13] studied divide-and-conquer methodology for high-dimensional partially linear models, focusing on the estimation and asymptotic distribution of the nonparametric function. Wang, Lian and Liang [21] proposed an additive partially linear framework for modelling massive heterogeneous data. Wang, Zhang and Lian [22] considered divide-and-conquer strategy for partially linear additive models with a high dimensional linear part.

Statistical modeling for massive data has attracted a lot of recent research. However, as far as we are aware, semiparametric inference for massive data, such as the single index regression model, the semiparametric varying-coefficient linear model, the partially linear errors-in-variables model, etc, still remains untouched. Motivated by the divide-and-conquer approach for generalized linear models with massive data in Chen and Xie [2], we consider a split-and-conquer variable selection procedure for high-dimensional general semiparametric models with massive data, focusing on selection consistency and oracle property of the proposed procedure. The main contributions of this paper are:

1. We propose a general procedure for variable selection in high-dimensional semiparametric models by penalized semiparametric estimating equations, including partially linear models, single index regression models, semiparametric varying-coefficient linear models, partially linear errors-in-variables models, etc. Under some regularity conditions, the oracle property is established, which the number of parameters is allowed to diverge.

2. For extraordinarily large data, we propose a split-and-conquer variable selection procedure for general semiparametric models based on penalized semiparametirc estimating functions. Under some regularity conditions, we establish the oracle property of the aggregated estimator $\check{\theta }$ when the number of subsets does not grow too fast. What is more, the split-and-conquer procedure enjoys the oracle property as the penalized estimator by using all the dataset, and can substantially reduce computing time and computer memory requirements. The main differences between the second main contribution in this paper and theirs we compare with are: (1) Zhao, Cheng and Liu [25], Lian, Zhao and Lv [13] and Wang, Lian and Liang [21] considered partially linear models with massive heterogeneous data, while we consider general semiparametric models with massive data by using the semiparametric estimating equations, including partially linear models, semiparametric varying-coefficient linear models, partially linear errors-in-variables models, single index regression models, etc. (2) Lian, Zhao and Lv [13], Wang, Lian and Liang [21] and Wang, Zhang and Lian [22] proposed to employ the polynomial splines to approximate the nonparametric functions, while we propose to estimate $H\left(T\right)$ by the kernel estimators.

The remainder of this paper is organized as follows. In Section 2, we propose a split-and-conquer variable selection procedure for high-dimensional general semiparametric models with massive data, and establish selection consistency and the oracle property of the proposed method. In Section 3, we propose a computational algorithm to solve the penalized semiparametric estimating equations. Simulation results are reported in Section 4, and one real data example is presented in Section 5. Finally, the technical proofs of main results are stated in the Appendix.