Skip to main content
SpringerLink
Log in

An inexact interior point method for L 1-regularized sparse covariance selection

  • Full Length Paper
  • Published:
Mathematical Programming Computation Aims and scope Submit manuscript

Abstract

Sparse covariance selection problems can be formulated as log-determinant (log-det) semidefinite programming (SDP) problems with large numbers of linear constraints. Standard primal–dual interior-point methods that are based on solving the Schur complement equation would encounter severe computational bottlenecks if they are applied to solve these SDPs. In this paper, we consider a customized inexact primal–dual path-following interior-point algorithm for solving large scale log-det SDP problems arising from sparse covariance selection problems. Our inexact algorithm solves the large and ill-conditioned linear system of equations in each iteration by a preconditioned iterative solver. By exploiting the structures in sparse covariance selection problems, we are able to design highly effective preconditioners to efficiently solve the large and ill-conditioned linear systems. Numerical experiments on both synthetic and real covariance selection problems show that our algorithm is highly efficient and outperforms other existing algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Banerjee O., El Ghaoui L., d’Aspremont A.: Model selection through sparse maximum likelihood estimation for multivariate Gaussian or binary data. J. Mach. Learn. Res. 9, 485–516 (2008)

    MathSciNet  Google Scholar 

  2. Bilmes, J.A.: Natural statistical models for automatic speech recognition. PhD thesis, University of California, Berkeley (1999)

  3. Burer S., Monteiro R.D.C., Zhang Y.: A computational study of a gradient-based log-barrier algorithm for a class of large-scale SDPs. Math. Program. 95, 359–379 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  4. Chen, S.S., Gopinath, R.A.: Model selection in acoustic modeling. In: Proc. EUROSPEECH’99, pp. 1087–1090, Budapest, Hungary (1999)

  5. Dahl J., Vandenberghe L., Roychowdhury V.: Covariance selection for nonchordal graphs via chordal embedding. Optim. Methods Softw. 23, 501–520 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  6. d’Aspremont, A.: Identifying small mean reverting portfolios. Quant. Finance (2010, to appear)

  7. d’Aspremont A., Banerjee O., El Ghaoui L.: First-order methods for sparse covariance selection. SIAM J. Matrix Anal. Appl. 30, 56–66 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  8. Dempster A.P.: Covariance selection. Biometrics 28, 157–175 (1972)

    Article  Google Scholar 

  9. Dobra A.: Variable selection and dependency networks for genomewide data. Biostatistics 10, 621–639 (2009)

    Article  Google Scholar 

  10. Edwards D.: Introduction to graphical modelling, 2nd edn. Springer, New York (2000)

    MATH  Google Scholar 

  11. Fan J., Feng Y., Wu Y.: Network exploration via the adaptive LASSO and SCAD penalties. Ann. Appl. Stat. 3, 521–541 (2009)

    Article  MATH  Google Scholar 

  12. Freund, R., Nachtigal, N.: A new Krylov-subspace method for symmetric indefinite linear system. In: Proceedings of the 14th IMACS World Congress on Computational and Applied Mathematics, Atlanta, USA, pp. 1253–1256 (1994)

  13. Friedman J., Hastie T., Tibshirani R.: Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9, 432–441 (2008)

    Article  MATH  Google Scholar 

  14. Golub T.R., Slonim D.K., Tamayo P., Huard C., Gaasenbeek M., Mesirov J.P., Coller H., Loh M.L., Downing J.R., Caligiuri M.A., Bloomfield C.D.: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286, 531–537 (1999)

    Article  Google Scholar 

  15. Hedenfalk I., Duggan D., Chen Y., Radmacher M., Bittner M., Simon R., Meltzer P., Gusterson B., Esteller M., Raffeld M., Yakhini Z., Ben-Dor A., Dougherty E., Kononen J., Bubendorf L., Fehrle W., Pittaluga S., Gruvberger S., Loman N., Johannsson O., Olsson H., Wilfond B., Sauter G., Kallioniemi O.-P., Borg A., Trent J.: Gene-expression profiles in hereditary breast cancer. N. Engl. J. Med. 344, 539–548 (2001)

    Article  Google Scholar 

  16. Jarre F., Rendl F.: An augmented primal-dual method for linear conic programs. SIAM J. Optim. 19, 808–823 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  17. Krishnamurthy, V., d’Aspremont, A.: A pathwise algorithm for covariance selection. Preprint (2009)

  18. Lan, G., Lu, Z., Monterio, R.D.: Primal-dual first-order methods with \({\mathcal{O}(1/\epsilon)}\) iteration-complexity for cone programming. Math. Program. (2010, to appear)

  19. Lauritzen, S.L.: Graphical models. In: Oxford Statistical Science Series, vol. 17. The Clarendon Press/Oxford University Press/Oxford Science Publications, New York (1996)

  20. Lu Z.: Smooth optimization approach for sparse covariance selection. SIAM J. Optim. 19, 1807–1827 (2008)

    Article  MathSciNet  Google Scholar 

  21. Lu Z.: Adaptive first-order methods for general sparse inverse covariance selection. SIAM J. Matrix Anal. Appl. 31, 2000–2016 (2010)

    Article  MATH  Google Scholar 

  22. Meinshausen N., Bühlmann P.: High-dimensional graphs and variable selection with the lasso. Ann. Stat. 34, 1436–1462 (2006)

    Article  MATH  Google Scholar 

  23. Nesterov Y.: Smooth minimization of non-smooth functions. Math. Program. 103, 127–152 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  24. Nesterov Y., Todd M.J.: Primal-dual interior-point methods for self-scaled cones. SIAM J. Optim. 8, 324–364 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  25. Pittman J., Huang E., Dressman H., Horng C.-F., Cheng S.H., Tsou M.-H., Chen C.-M., Bild A., Iversen E.S., Huang A.T., Nevins J.R., West M.: Integrated modeling of clinical and gene expression information for personalized prediction of disease outcomes. Proc. Natl. Acad. Sci. USA 101(22), 8431–8436 (2004)

    Article  Google Scholar 

  26. Rockafellar R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1, 97–116 (1976)

    Article  MATH  MathSciNet  Google Scholar 

  27. Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. Society for Industrial and Applied Mathematics, Philadelphia (2003)

  28. Sachs K., Perez O., Pe’er D., Lauffenburger D.A., Nolan G.P.: Causal protein-signaling networks derived from multiparameter single-cell data. Science 308, 523–529 (2005)

    Article  Google Scholar 

  29. Scheinberg, K., Rish, I.: Learning sparse Gaussian Markov networks using a greedy coordinate ascent approach. In: Balcázar, J., Bonchi, F., Gionis, A., Sebag, M. (eds.) Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science 6323. pp. 196–212 (2010)

  30. Storey J.D., Tibshirani R.: Statistical significance for genome-wide studies. Proc. Natl. Acad. Sci. USA 100(16), 9440–9445 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  31. Sturm J.F.: Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones. Optim. Methods Softw. 11/12, 625–653 (1999)

    Article  MathSciNet  Google Scholar 

  32. Toh K.-C.: Solving large scale semidefinite programs via an iterative solver on the augmented systems. SIAM J. Optim. 14, 670–698 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  33. Toh K.-C.: An inexact primal-dual path following algorithm for convex quadratic SDP. Math. Program. 112, 221–254 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  34. Toh K.-C., Todd M.J., Tütüncü R.H.: SDPT3—a MATLAB software package for semidefinite programming, version 1.3. Optim. Methods Softw. 11/12, 545–581 (1999)

    Article  Google Scholar 

  35. Tsuchiya T., Xia Y.: An extension of the standard polynomial-time primal-dual path-following algorithm to the weighted determinant maximization problem with semidefinite constraints. Pac. J. Optim. 3, 165–182 (2007)

    MATH  MathSciNet  Google Scholar 

  36. Ueno U., Tsuchiya T.: Covariance regularization in inverse space. Q. J. R. Meteorol. Soc. 135, 1133–1156 (2009)

    Article  Google Scholar 

  37. Vandenberghe L., Boyd S., Wu S.-P.: Determinant maximization with linear matrix inequality constraints. SIAM J. Matrix Anal. Appl. 19, 499–533 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  38. Wang C., Sun D., Toh K.-C.: Solving log-determinant optimization problems by a newton-cg proximal point algorithm. SIAM J. Optim. 20, 2994–3013 (2010)

    Article  Google Scholar 

  39. Whittaker, J.: Graphical models in applied multivariate statistics. In: Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics. Wiley, Chichester (1990)

  40. Wille, A., Zimmermann, P., Vranová, E., Fürholz, A., Laule, O., Bleuler, S., Hennig, L., Prelić, A., von Rohr, P., Thiele, L., Zitzler, E., Gruissem, W., Bühlmann, P.: Sparse graphical gaussian modeling of the isoprenoid gene network in Arabidopsis thaliana. Genome Biol. 5, R92 (2004)

  41. Wong F., Carter C.K., Kohn R.: Efficient estimation of covariance selection models. Biometrika 90, 809–830 (2003)

    Article  MathSciNet  Google Scholar 

  42. Wu W.B., Pourahmadi M.: Nonparameteric estimation of large covariance matrices of longitudinal data. Biometrika 90, 831–844 (2003)

    Article  MathSciNet  Google Scholar 

  43. Yeung K.Y., Bumgarner R.E., Raftery A.E.: Bayesian model averaging: development of an improved multi-class, gene selection and classification tool for microarray data. Bioinformatics 21, 2394–2402 (2005)

    Article  Google Scholar 

  44. Yuan M., Lin Y.: Model selection and estimation in the Gaussian graphical model. Biometrika 94, 19–35 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  45. Yuan, X.: Alternating direction methods for sparse covariance selection. Preprint (2009)

  46. Zhang Y.: On extending some primal-dual interior-point algorithms from linear programming to semidefinite programming. SIAM J. Optim. 8, 365–386 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  47. Zhao X.Y., Sun D., Toh K.-C.: A Newton-CG augmented Lagrangian method for semidefinite programming. SIAM J. Optim. 20, 1737–1765 (2010)

    Article  MathSciNet  Google Scholar 

  48. Zhou G., Toh K.-C.: Polynomiality of an inexact infeasible interior point algorithm for semidefinite programming. Math. Program. 99, 261–282 (2004)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, National University of Singapore, 2 Science Drive 2, Singapore, 117543, Singapore

    Lu Li & Kim-Chuan Toh

  2. Singapore-MIT Alliance, 4 Engineering Drive 3, Singapore, 117576, Singapore

    Kim-Chuan Toh

Authors
  1. Lu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kim-Chuan Toh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kim-Chuan Toh.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Li, L., Toh, KC. An inexact interior point method for L 1-regularized sparse covariance selection. Math. Prog. Comp. 2, 291–315 (2010). https://doi.org/10.1007/s12532-010-0020-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12532-010-0020-6

Keywords

Mathematics Subject Classification (2000)

Search

Navigation