Efficient block-coordinate descent algorithms for the Group Lasso

Abstract

We present two algorithms to solve the Group Lasso problem (Yuan and Lin in, J R Stat Soc Ser B (Stat Methodol) 68(1):49–67, 2006). First, we propose a general version of the Block Coordinate Descent (BCD) algorithm for the Group Lasso that employs an efficient approach for optimizing each subproblem exactly. We show that it exhibits excellent performance when the groups are of moderate size. For groups of large size, we propose an extension of ISTA/FISTA SIAM (Beck and Teboulle in, SIAM J Imag Sci 2(1):183–202, 2009) based on variable step-lengths that can be viewed as a simplified version of BCD. By combining the two approaches we obtain an implementation that is very competitive and often outperforms other state-of-the-art approaches for this problem. We show how these methods fit into the globally convergent general block coordinate gradient descent framework in Tseng and Yun (Math Program 117(1):387–423, 2009). We also show that the proposed approach is more efficient in practice than the one implemented in Tseng and Yun (Math Program 117(1):387–423, 2009). In addition, we apply our algorithms to the Multiple Measurement Vector (MMV) recovery problem, which can be viewed as a special case of the Group Lasso problem, and compare their performance to other methods in this particular instance.

Appendices

Appendix A: Simulated Group Lasso data sets

For the specifications of the data sets in Table 2, interested readers can refer to the references provided at the beginning of Sect. 7.1. Here, we provide the details of the data sets in Table 3.

1.1 A.1 yl1L

50 latent variables $Z_{1}, \ldots , Z_{50}$ are simulated from a centered multivariate Gaussian distribution with covariance between $Z_{i}$ and $Z_{j}$ being $0.5^{|i-j|}$. The first 47 latent variables are encoded in $\{0,\ldots ,9\}$ according to their inverse cdf values as done in [33]. The last three variables are encoded in $\{0,\ldots ,999\}$. Each latent variable corresponds to one segment and contributes $L$ columns in the design matrix with each column $j$ containing values of the indicator function $I(Z_{i}=j)$. $L$ is the size of the encoding set for $Z_{i}$. The responses are a linear combination of a sparse selection of the segments plus a Gaussian noise. We simulate 5,000 observations.

1.2 A.2 yl4L

110 latent variables are simulated in the same way as the third data set in [33]. The first 50 variables contribute 3 columns each in the design matrix $A$ with the $i$-th column among the three containing the $i$-th power of the variable. The next 50 variables are encoded in a set of 3, and the final 10 variables are encoded in a set of 50, similar to yl1L. In addition, 4 groups of 1,000 Gaussian random numbers are also added to $A$. The responses are constructed in a similar way as in yl1L. 2,000 observation are simulated.

1.3 A.3 mgb2L

5,001 variables are simulated as in yl1L without categorization. They are then divided into six groups, with first containing one variable and the rest containing 1,000 each. The responses are constructed in a similar way as in yl1L. We collect 2,000 observations.

1.4 A.4 ljyL

We simulate 15 groups independent standard Gaussian random variables. The first five groups are of a size 5 each, and the last 10 groups contain 500 variables each. The responses are constructed in a similar way as in yl1L, and we simulate 2,000 observations.

1.5 A.5 glassoL1, glassoL2

The design matrix $A$ is drawn from the standard Gaussian distribution. The length of each group is sampled from the uniform distribution $U(10,50)$ with probability 0.9 and from the uniform distribution $U(50,300)$ with probability 0.1. We continue to generate the groups until the specified number of features $m$ is reached. The ground truth feature coefficients $x^*$ is generated from $\mathcal N (2,4)$ with approximately $5~\%$ non-zero values, and we assume a $\mathcal N (0.5,0.25)$ noise.

Appendix B: MMV scalability test sets

The data sets in Table 5 are generated in the same say with different attributes. Both the design matrix $A$ and the non-zeros rows in the ground truth $X_0$ are drawn from the standard Gaussian distribution. The indices of the non-zero rows in $X_0$ are uniformly sampled. The measurement matrix $B$ is obtained by $B = AX_0$. The data attributes, such as the number of measurements and the sparsity level of the ground truth, are set to ensure that the exact solution is recoverable by the MMV model.