Abstract
We consider a class of unconstrained nonsmooth convex optimization problems, in which the objective function is the sum of a convex smooth function on an open subset of matrices and a separable convex function on a set of matrices. This problem includes the covariance selection problem that can be expressed as an ℓ 1-penalized maximum likelihood estimation problem. In this paper, we propose a block coordinate gradient descent method (abbreviated as BCGD) for solving this class of nonsmooth separable problems with the coordinate block chosen by a Gauss-Seidel rule. The method is simple, highly parallelizable, and suited for large-scale problems. We establish global convergence and, under a local Lipschizian error bound assumption, linear rate of convergence for this method. For the covariance selection problem, the method can terminate in \({O(n^3/\epsilon)}\) iterations with an \({\epsilon}\)-optimal solution. We compare the performance of the BCGD method with the first-order methods proposed by Lu (SIAM J Optim 19:1807–1827, 2009; SIAM J Matrix Anal Appl 31:2000–2016, 2010) for solving the covariance selection problem on randomly generated instances. Our numerical experience suggests that the BCGD method can be efficient for large-scale covariance selection problems with constraints.
Similar content being viewed by others
References
Borwein J.M., Lewis A.S.: Convex Analysis and Nonlinear Optimization: Theory and Examples. Springer, New York (2000)
Dahl J., Vandenberghe L., Roychowdhury V.: Covariance selection for non-chordal graphs via chordal embedding. Optim. Methods Softw. 23, 501–520 (2008)
D’Aspremont A., Banerjee O., El Ghaoui L.: First-order methods for sparse covariance selection. SIAM J. Matrix Anal. Appl. 30, 56–66 (2008)
Friedman J., Hastie T., Tibshirani R.: Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9, 432–441 (2008)
Goebel R., Rockafellar R.T.: Local strong convexity and local Lipschitz continuity of the gradients of convex functions. J. Convex Anal. 15, 263–270 (2008)
Horn R., Johnson C.: Matrix Analysis. Cambridge University Press, Cambridge (1999)
Lu Z.: Smooth optimization approach for covariance selection. SIAM J. Optim. 19, 1807–1827 (2009)
Lu Z.: Adaptive first-order methods for general sparse inverse covariance selection. SIAM J. Matrix Anal. Appl. 31, 2000–2016 (2010)
Nesterov Y.: A method of solving a convex programming problem with convergence rate O(1/k 2). Sov. Math. Doklady 27, 372–376 (1983)
Nesterov Y.: Smooth minimization of nonsmooth functions. Math. Prog. 103, 127–152 (2005)
Nocedal J., Wright S.J.: Numerical Optimization. Springer, New York (1999)
Rockafellar R.T.: Convex Analysis. Princeton University Press, Princeton (1970)
Rockafellar R.T., Wets R.J.-B.: Variational Analysis. Springer, New York (1998)
Saad Y.: Iterative Methods for Sparse Linear Systems. PWS Publishing Company, Boston (1996)
Scheinberg, K., Rish, I.: Learning sparse Gaussian markov networks using a greedy coordinate ascent approach. In: Machine Learning and Knowledge Discovery in Databases. Lecture Notes in Computer Science, vol. 6323, pp. 196–212 (2010)
Tseng P.: Convergence of block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl. 109, 473–492 (2001)
Tseng P., Yun S.: A coordinate gradient descent method for nonsmooth separable minimization. Math. Prog. Ser. B 117, 387–423 (2009)
Wang C., Sun D., Toh K.-C.: Solving log-determinant optimization problems by a Newton-CG primal proximal point algorithm. SIAM J. Optim. 20, 2994–3013 (2010)
Wong F., Carter C.K., Kohn R.: Efficient estimation of covariance selection models. Biometrika 90, 809–830 (2003)
Yuan, X.: Alternating direction methods for sparse covariance selection. http://www.optimization-online.org/DB_FILE/2009/09/2390.pdf
Author information
Authors and Affiliations
Corresponding author
Additional information
In memory of Paul Tseng, who went missing while on a kayaking trip along the Jinsha river, China, on August 13, 2009, for his friendship and contributions to large-scale continuous optimization.
Rights and permissions
About this article
Cite this article
Yun, S., Tseng, P. & Toh, KC. A block coordinate gradient descent method for regularized convex separable optimization and covariance selection. Math. Program. 129, 331–355 (2011). https://doi.org/10.1007/s10107-011-0471-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-011-0471-1
Keywords
- Block coordinate gradient descent
- Complexity
- Convex optimization
- Covariance selection
- Global convergence
- Linear rate convergence
- ℓ1-penalization
- Maximum likelihood estimation