Skip to main content
SpringerLink
Log in
Book cover

International Conference on Learning and Intelligent Optimization

LION 2020: Learning and Intelligent Optimization pp 264–277Cite as

An Alternating DCA-Based Approach for Reduced-Rank Multitask Linear Regression with Covariance Estimation

  • Conference paper
  • First Online:

  • 848 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12096))

Abstract

We investigate a nonconvex, nonsmooth optimization approach based on DC (Difference of Convex functions) programming and DCA (DC Algorithm) for the reduced-rank multitask linear regression problem with covariance estimation. The objective is to model the linear relationship between a multitask response and more explanatory variables by estimating a low-rank coefficient matrix and a covariance matrix. The problem is formulated as minimizing the constrained negative log-likelihood function of these two matrix variables. Then, we consider a reformulation of this problem which takes the form of a partial DC program i.e. it is a standard DC program for each variable when fixing the other variable. Next, an alternating version of a standard DCA scheme is developed. Numerical results on many synthetic multitask linear regression datasets and benchmark real datasets show the efficiency of our approach in comparison with the existing alternating/joint methods.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    For the detailed descriptions of all datasets, the reader is referred to [37] and the website http://mulan.sourceforge.net/datasets-mtr.html.

References

  1. Aldrin, M.: Reduced-Rank Regression, vol. 3, pp. 1724–1728. Wiley, Hoboken (2002)

    Google Scholar 

  2. Chen, L., Huang, J.Z.: Sparse reduced-rank regression with covariance estimation. Stat. Comput. 461–470 (2014). https://doi.org/10.1007/s11222-014-9517-6

  3. Cover, T.M., Thomas, A.: Determinant inequalities via information theory. SIAM J. Matrix Anal. Appl. 9(3), 384–392 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  4. Dev, H., Sharma, N.L., Dawson, S.N., Neal, D.E., Shah, N.: Detailed analysis of operating time learning curves in robotic prostatectomy by a novice surgeon. BJU Int. 109(7), 1074–1080 (2012)

    Article  Google Scholar 

  5. Duník, J., Straka, O., Kost, O., Havlík, J.: Noise covariance matrices in state-space models: a survey and comparison of estimation methods - part i. Int. J. Adapt. Control Sig. Process. 31(11), 1505–1543 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  6. Eckart, C., Young, G.: The approximation of one matrix by another of lower rank. Psychometrika 1, 211–218 (1936)

    Article  MATH  Google Scholar 

  7. Ha, W., Foygel Barber, R.: Alternating minimization and alternating descent over nonconvex sets. ArXiv e-prints, September 2017

    Google Scholar 

  8. Harrison, L., Penny, W., Friston, K.: Multivariate autoregressive modeling of fMRI time series. NeuroImage 19, 1477–1491 (2003)

    Google Scholar 

  9. He, D., Parida, L., Kuhn, D.: Novel applications of multitask learning and multiple output regression to multiple genetic trait prediction. Bioinformatics 32(12), i37–i43 (2016)

    Article  Google Scholar 

  10. Hyams, E., Mullins, J., Pierorazio, P., Partin, A., Allaf, M., Matlaga, B.: Impact of robotic technique and surgical volume on the cost of radical prostatectomy. J. Endourol. 27(3), 298–303 (2013)

    Article  Google Scholar 

  11. Ioffe, A., Tihomirov, V.: Theory of Extremal Problems. North-Holland (1979)

    Google Scholar 

  12. Le Thi, H.A.: Analyse numérique des algorithmes de l’optimisation d. C. Approches locale et globale. Codes et simulations numériques en grande dimension. Applications. Ph.D. thesis, University of Rouen (1994)

    Google Scholar 

  13. Le Thi, H.A.: Collaborative DCA: an intelligent collective optimization scheme, and its application for clustering. J. Intell. Fuzzy Syst. 37(6), 7511–7518 (2019)

    Article  Google Scholar 

  14. Le Thi, H.A.: DC programming and DCA for supply chain and production management: state-of-the-art models and methods. Int. J. Prod. Res. 1–37 (2019). https://doi.org/10.1080/00207543.2019.1657245

  15. Le Thi, H.A., Ho, V.T.: Online learning based on online DCA and application to online classification. Neural Comput. 32(4), 759–793 (2020)

    Article  Google Scholar 

  16. Le Thi, H.A., Ho, V.T., Pham Dinh, T.: A unified DC programming framework and efficient DCA based approaches for large scale batch reinforcement learning. J. Glob. Optim. 73(2), 279–310 (2018). https://doi.org/10.1007/s10898-018-0698-y

    Article  MathSciNet  MATH  Google Scholar 

  17. Le Thi, H.A., Huynh, V.N., Pham Dinh, T.: Alternating DC algorithm for partial DC programming (2016). Technical report, University of Lorraine

    Google Scholar 

  18. Le Thi, H.A., Nguyen, M.C.: Self-organizing maps by difference of convex functions optimization. Data Mining Knowl. Discov. (2), 1336–1365 (2014). https://doi.org/10.1007/s10618-014-0369-7

  19. Le Thi, H.A., Pham Dinh, T.: The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems. Ann. Oper. Res. 133(1–4), 23–46 (2005)

    MathSciNet  MATH  Google Scholar 

  20. Le Thi, H.A., Pham Dinh, T.: Difference of convex functions algorithms (DCA) for image restoration via a Markov random field model. Optim. Eng. 18(4), 873–906 (2017). https://doi.org/10.1007/s11081-017-9359-0

    Article  MathSciNet  MATH  Google Scholar 

  21. Le Thi, H.A., Pham Dinh, T.: DC programming and DCA: thirty years of developments. Math. Program. 169(1), 5–68 (2018). https://doi.org/10.1007/s10107-018-1235-y

    Article  MathSciNet  MATH  Google Scholar 

  22. Le Thi, H.A., Pham Dinh, T., Le, H.M., Vo, X.T.: DC approximation approaches for sparse optimization. Eur. J. Oper. Res. 244(1), 26–46 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  23. Le Thi, H.A., Pham Dinh, T., Ngai, H.V.: Exact penalty and error bounds in DC programming. J. Glob. Optim. 52(3), 509–535 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  24. Le Thi, H.A., Ta, A.S., Pham Dinh, T.: An efficient DCA based algorithm for power control in large scale wireless networks. Appl. Math. Comput. 318, 215–226 (2018)

    MathSciNet  MATH  Google Scholar 

  25. Lee, C.L., Lee, J.: Handbook of Quantitative Finance and Risk Management. Springer, Heidelberg (2010). https://doi.org/10.1007/978-0-387-77117-5

  26. Lee, W., Liu, Y.: Simultaneous multiple response regression and inverse covariance matrix estimation via penalized Gaussian maximum likelihood. J. Multivariate Anal. 111, 241–255 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  27. Nez, A., Fradet, L., Marin, F., Monnet, T., Lacouture, P.: Identification of noise covariance matrices to improve orientation estimation by Kalman filter. Sensors 18, 3490 (2018)

    Google Scholar 

  28. Ong, C.S., Le Thi, H.A.: Learning sparse classifiers with difference of convex functions algorithms. Optim. Methods Softw. 28(4), 830–854 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  29. Ortega, J., Rheinboldt, W.: Iterative Solution of Nonlinear Equations in Several Variables. Academic Press (1970)

    Google Scholar 

  30. Pham Dinh, T., Le Thi, H.A.: Convex analysis approach to DC programming: theory, algorithms and applications. Acta Mathematica Vietnamica 22(1), 289–355 (1997)

    MathSciNet  MATH  Google Scholar 

  31. Pham Dinh, T., Le Thi, H.A.: DC optimization algorithms for solving the trust region subproblem. SIAM J. Optim. 8(2), 476–505 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  32. Pham Dinh, T., Le Thi, H.A.: Recent advances in DC programming and DCA. In: Nguyen, N.-T., Le-Thi, H.A. (eds.) Transactions on Computational Intelligence XIII. LNCS, vol. 8342, pp. 1–37. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54455-2_1

    Chapter  Google Scholar 

  33. Phan, D.N., Le Thi, H.A., Pham Dinh, T.: Sparse covariance matrix estimation by DCA-based algorithms. Neural Comput. 29(11), 3040–3077 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  34. Rothman, A.J., Levina, E., Zhu, J.: Sparse multivariate regression with covariance estimation. J. Comput. Graph. Stat. 19(4), 947–962 (2010)

    Article  MathSciNet  Google Scholar 

  35. Smith, A.E., Coit, D.W.: Constraint-handling techniques - penalty functions. In: Handbook of Evolutionary Computation, pp. C5.2:1–C5.2.6. Oxford University Press (1997)

    Google Scholar 

  36. Sohn, K.A., Kim, S.: Joint estimation of structured sparsity and output structure in multiple-output regression via inverse-covariance regularization. In: Lawrence, N.D., Girolami, M. (eds.) Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 22, pp. 1081–1089. PMLR, La Palma (2012)

    Google Scholar 

  37. Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W., Vlahavas, I.: Multi-target regression via input space expansion: treating targets as inputs. Mach. Learn. 104(1), 55–98 (2016). https://doi.org/10.1007/s10994-016-5546-z

    Article  MathSciNet  MATH  Google Scholar 

  38. Tran, T.T., Le Thi, H.A., Pham Dinh, T.: DC programming and DCA for enhancing physical layer security via cooperative jamming. Comput. Oper. Res. 87, 235–244 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  39. Wold, S., Sjöström, M., Eriksson, L.: PLS-regression: a basic tool of chemometrics. Chem. Intell. Lab. Syst. 58(2), 109–130 (2001)

    Article  Google Scholar 

  40. Yuan, M., Lin, Y.: Model selection and estimation in the Gaussian graphical model. Biometrika 94(1), 19–35 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  41. Zălinescu, C.: Convex Analysis in General Vector Spaces. World Scientific (2002)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science and Applications Department, LGIPM, University of Lorraine, Metz, France

    Vinh Thanh Ho & Hoai An Le Thi

Authors
  1. Vinh Thanh Ho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hoai An Le Thi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vinh Thanh Ho .

Editor information

Editors and Affiliations

  1. CARGO Lab., Wilfrid Laurier University, Waterloo, ON, Canada

    Ilias S. Kotsireas

  2. Center for Applied Optimization, University of Florida, Gainesville, FL, USA

    Panos M. Pardalos

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ho, V.T., Le Thi, H.A. (2020). An Alternating DCA-Based Approach for Reduced-Rank Multitask Linear Regression with Covariance Estimation. In: Kotsireas, I., Pardalos, P. (eds) Learning and Intelligent Optimization. LION 2020. Lecture Notes in Computer Science(), vol 12096. Springer, Cham. https://doi.org/10.1007/978-3-030-53552-0_25

Download citation

Publish with us

Policies and ethics

Search

Navigation