Skip to main content
SpringerLink
Log in

On adaptive block coordinate descent methods for ridge regression

  • Published:
Computational and Applied Mathematics Aims and scope Submit manuscript

Abstract

For solving large-scale ridge regression problem, two adaptive block coordinate descent methods based on the greedy criterion are proposed. The first one called adaptive block Kaczmarz (ABK) method is designed for the underdetermined system while the second one called adaptive block Gauss–Seidel (ABGS) method is designed for the overdetermined system, where both of them do not need to predetermine block pavings or solve any subsystems. The convergence theories are established, and the numerical experiments are conducted. Numerical results illustrate that both ABK and ABGS significantly outperform the state-of-art methods especially in the sense of computing times.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abbas SH (2004) A parallel algorithm of block Gauss–Seidel method for solving linear system of equations. Int J Comput Numer Anal Appl 5:339–347

    MathSciNet  MATH  Google Scholar 

  • Bai ZZ, Wu WT (2018a) On greedy randomized Kaczmarz method for solving large sparse linear systems. SIAM J Sci Comput 40:A592–A606

  • Bai ZZ, Wu WT (2018b) On relaxed greedy randomized Kaczmarz methods for solving large sparse linear systems. Appl Math Lett 83:21–26

  • Bai ZZ, Wu WT (2019) On greedy randomized coordinate descent methods for solving large linear least-squares problems. Numer Linear Algebra Appl 26:e2237

    MathSciNet  MATH  Google Scholar 

  • Bai ZZ, Wu WT (2019) On partially randomized extended Kaczmarz method for solving large sparse overdetermined inconsistent linear systems. Linear Algebra Appl 578:225–250

    MathSciNet  MATH  Google Scholar 

  • Bai ZZ, Wu WT (2021) On greedy randomized augmented Kaczmarz method for solving large sparse inconsistent linear systems. SIAM J Sci Comput 43:A3892-3911

    MathSciNet  MATH  Google Scholar 

  • Bai ZZ, Wang L, Muratova GV (2022) On relaxed greedy randomized augmented Kaczmarz methods for solving large sparse inconsistent linear systems. East Asian J Appl Math 12:323–332

    MathSciNet  MATH  Google Scholar 

  • Chen JQ, Huang ZD (2022a) On a fast deterministic block Kaczmarz method for solving large-scale linear systems. Numer Algorithms 89:1007–1029

  • Chen JQ, Huang ZD (2022b) A fast block coordinate descent method for solving linear least-squares problems. East Asian J Appl Math 12:406–420

  • Davis TA, Hu Y (2011) The University of Florida sparse matrix collection. ACM Trans Math Softw 38:1–25

    MathSciNet  MATH  Google Scholar 

  • Duan LX, Zhang GF (2021) Variant of greedy randomized Gauss–Seidel method for ridge regression. Numer Math Theor Methods Appl 14:714–737

    MathSciNet  MATH  Google Scholar 

  • Elfving T (1980) Block-iterative methods for consistent and inconsistent linear equations. Numer Math 35:1–12

    MathSciNet  MATH  Google Scholar 

  • Golub GH, Heath MT, Wahba G (1979) Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21:215–223

    MathSciNet  MATH  Google Scholar 

  • Gower RM, Richtarik P (2015) Randomized iterative methods for linear systems. SIAM J Matrix Anal Appl 36:1660–1690

    MathSciNet  MATH  Google Scholar 

  • Hadgu A (1984) An application of ridge regression analysis in the study of syphilis data. Stat Med 3:293–299

    Google Scholar 

  • Hansen PC (1992) Analysis of discrete ill-posed problems by means of the L-curve. SIAM Rev 34:561–580

    MathSciNet  MATH  Google Scholar 

  • Hawkins DM, Yin X (2002) A faster algorithm for ridge regression of reduce rank data. Comput Stat Data Anal 40:253–262

    MATH  Google Scholar 

  • Hefny A, Needell D, Ramdas A (2017) Rows versus columns: randomized Kaczmarz or Gauss–Seidel for ridge regression. SIAM J Sci Comput 39:S528–S542

    MathSciNet  MATH  Google Scholar 

  • Ivanov AA, Zhdanov AI (2013) Kaczmarz algorithm for Tikhonov regularization problem. Appl Math E-Notes 13:270–276

    MathSciNet  MATH  Google Scholar 

  • Jiang XL, Zhang K, Yin JF (2022) Randomized block Kaczmarz methods with k-means clustering for solving large linear systems. J Comput Appl Math 403:113828

    MathSciNet  MATH  Google Scholar 

  • Kaczmarz S (1937) Angenäherte auflösung von systemin linearer gleichungen. Bull Int Acad Pol Sci Lett A35:355–357

    MATH  Google Scholar 

  • Lei J, Liu S, Li ZH, Sun M (2009) An image reconstruction algorithm based on the extended Tikhonov regularization method for electrical capacitance tomography. Measurement 42:368–376

    Google Scholar 

  • Leventhal D, Lewis AS (2010) Randomized methods for linear constraints: convergence rates and conditioning. Math Oper Res 35:641–654

    MathSciNet  MATH  Google Scholar 

  • Li HY, Zhang YJ (2020) Greedy block Gauss–Seidel methods for solving large linear least-squares problem (2020). arxiv.org/abs/2004.02476

  • Li SX, Wang HX, Liu TH, Cui ZQ, Chen JN, Xia ZH, Guo Q (2022) A fast Tihonov regularization method based on homotopic mapping for electrical resistance tomography. Rev Sci Instrum 93:043709

    Google Scholar 

  • Liu Y, Gu CQ (2019) Variant of greedy randomized Kaczmarz for ridge regression. Appl Numer Math 143:223–246

    MathSciNet  MATH  Google Scholar 

  • Ma A, Needell D, Randas A (2015) Convergence properties of the randomized extended Gauss–Seidel and Kaczmarz methods. SIAM J Matrix Anal Appl 36:1590–1604

    MathSciNet  MATH  Google Scholar 

  • Moorman JD, Tu TK, Molitor D, Needell D (2021) Randomized Kaczmarz with averaging. BIT Numer Math 61:337–359

    MathSciNet  MATH  Google Scholar 

  • Morozov VA (1984) Methods for solving incorrectly posed problems. Springer, New York

    Google Scholar 

  • Necoara I (2019) Faster randomized block Kaczmarz algorithms. SIAM J Matrix Anal Appl 40:1425–1452

    MathSciNet  MATH  Google Scholar 

  • Needell D (2010) Randomized Kaczmarz solver for noisy linear systems. BIT Numer Math 50:395–403

    MathSciNet  MATH  Google Scholar 

  • Needell D, Tropp JA (2014) Paved with good intentions: analysis of a randomized block Kaczmarz method. Linear Algebra Appl 441:199–221

    MathSciNet  MATH  Google Scholar 

  • Needell D, Zhao R, Zouzias A (2015) Randomized block Kaczmarz method with projection for solving least squares. Linear Algebra Appl 484:322–343

    MathSciNet  MATH  Google Scholar 

  • Niu YQ, Zheng B (2020) A greedy block Kaczmarz algorithm for solving large-scale linear systems. Appl Math Lett 104:106294

    MathSciNet  MATH  Google Scholar 

  • Park C, Park H (2005) A relationship between linear discriminant analysis and the generalized minimum squared error solution. SIAM J Matrix Anal Appl 27:474–492

    MathSciNet  MATH  Google Scholar 

  • Popa C, Zdunek R (2004) Kaczmarz extended algorithm for tomographic image reconstruction from limited-data. Math Comput Simul 65:579–598

    MathSciNet  Google Scholar 

  • Rebrova E, Needell D (2021) On block Gaussian sketching for the Kaczmarz method. Numer Algorithms 86:443–473

    MathSciNet  MATH  Google Scholar 

  • Strohmer T, Vershynin R (2009) A randomized Kaczmarz algorithm with exponentail convergence. J Fourier Anal Appl 15:262–278

    MathSciNet  MATH  Google Scholar 

  • Tikhonov AN (1963) Solution of incorrectly formulated problems and the regularization method. Sov Math 4:1035–1038

    MATH  Google Scholar 

  • Wei W, Dai H, Liang WT (2020) Regularized least squares locality preserving projections with applications to image recognition. Neural Netw 128:322–330

    MATH  Google Scholar 

  • Wright SJ (2015) Coordinate descent methods. Math Program 151:3–34

    MathSciNet  MATH  Google Scholar 

  • Wu WM (2018) Convergence of the randomized block Gauss–Seidel method. https://doi.org/10.1137/17S015860

  • Wu NC, Xiang H (2022) On the generally randomized extended Gauss–Seidel method. Appl Numer Math 172:382–392

    MathSciNet  MATH  Google Scholar 

  • Wu NC, Cui LX, Zuo Q (2022) On the relaxed greedy deterministic row and column iterative methods. Appl Math Comput 432:127339

    MathSciNet  MATH  Google Scholar 

  • Yang X (2021) A geometric probability randomized Kaczmarz method for large scale linear systems. Appl Numer Math 164:139–160

    MathSciNet  MATH  Google Scholar 

  • Yang AL, Chen XQ (2022) A partially greedy randomized extended Gauss–Seidel method for solving large linear systems. East Asian J Appl Math 12:874–890

    MathSciNet  MATH  Google Scholar 

  • Ye ZS, Li JG, Zhang M (2014) Application of ridge regression and factor analysis in design and production of alloy wheels. J Appl Stat 41:1436–1452

    MathSciNet  MATH  Google Scholar 

  • Zhang JH, Guo JH (2020) On relaxed greedy randomized coordinate descent methods for solving large linear least-squares problems. Appl Numer Math 157:372–384

    MathSciNet  MATH  Google Scholar 

  • Zhang YJ, Li HY (2021) A count sketch maximal weighted residual Kaczmarz method for solving highly overdetermined linear systems. Appl Math Comput 410:126486

    MathSciNet  MATH  Google Scholar 

  • Zouzias A, Freris NM (2013) Randomized extended Kaczmarz for solving least squares. SIAM J Matrix Anal Appl 34:773–793

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors are grateful to the editor and the referees for their useful comments and suggestions which helped to improve the presentation of this paper. The research was supported by the National Natural Science Foundation of China (Grant Nos. 12201302 and 12001396), the Natural Science Foundation of Jiangsu Province (Grant Nos. BK20170591 and BK20200268), the Natural Science Foundation of Jiangsu Higher Education Institutions of China (Grant Nos. 21KJB110017 and 20KJB110005), the China Postdoctoral Science Foundation (Grant No. 2018M642130) and Qing Lan Project of the Jiangsu Higher Education Institutions.

Author information

Authors and Affiliations

  1. Department of Applied Mathematics, Nanjing Forestry University, Nanjing, 210037, People’s Republic of China

    Wei Wei & Tao Shi

  2. School of Mathematics, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, People’s Republic of China

    Song Nie

  3. Department of Mathematics, Taizhou University, Taizhou, 225300, People’s Republic of China

    Xiaoping Chen

  4. Department of Mathematics, Southeast University, Nanjing, 210096, People’s Republic of China

    Xiaoping Chen

Authors
  1. Wei Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tao Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Song Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoping Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoping Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Communicated by Margherita Porcelli.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wei, W., Shi, T., Nie, S. et al. On adaptive block coordinate descent methods for ridge regression. Comp. Appl. Math. 42, 315 (2023). https://doi.org/10.1007/s40314-023-02453-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s40314-023-02453-0

Keywords

Mathematics Subject Classification

Search

Navigation