Skip to main content
SpringerLink
Log in

A semi-randomized Kaczmarz method with simple random sampling for large-scale linear systems

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

Randomized Kaczmarz-type methods are appealing for large-scale linear systems arising from big data problems. One of the keys of randomized Kaczmarz-type methods is how to effectively select working rows from the coefficient matrix. To the best of our knowledge, most of the randomized Kaczmarz-type methods need to compute probabilities for choosing working rows. However, when the amount of data is huge, the computation of probabilities will be inaccurate due to many factors such as rounding errors and data distribution. Moreover, in some popular randomized Kaczmarz methods, we have to scan all the rows of the data matrix in advance, or to compute residual of the linear system in each step. Hence, we have to access all the rows of the data matrix, which are unfavorable for big data problems. To overcome these difficulties, we first introduce a semi-randomized Kaczmarz method in which there is no need to compute probabilities explicitly. However, we still have to access all the rows of the matrix for the computation of residuals. To improve the semi-randomized Kaczmarz method further, inspired by Chebyshev’s (weak) law of large numbers, we apply the simple sampling strategy to the semi-randomized Kaczmarz method, and propose a semi-randomized Kaczmarz method with simple random sampling. In the new method, there is no need to calculate probabilities explicitly and it is free of computing residuals of the linear system, and is free of constructing index sets via scanning residuals. Indeed, we only need to compute some elements of residuals corresponding to simple sampling sets, and a small portion of rows of the matrix are utilized. Convergence results are established to show the rationality and feasibility of the two proposed methods. Numerical experiments demonstrate the superiority of the new methods over many state-of-the-art randomized Kaczmarz methods for large-scale linear systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bai, Z.-Z., Liu, X.-G.: On the Meany inequality with applications to convergence analysis of several row-action iteration methods. Numer. Math. 124, 215–236 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  2. Bai, Z.-Z., Wu, W.-T.: On convergence rate of the randomized Kaczmarz method. Linear Algebra Appl 553, 252–269 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bai, Z.-Z., Wu, W.-T.: On greedy randomized Kaczmarz method for solving large sparse linear systems. SIAM J. Sci. Comput. 40, A592–A606 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bai, Z.-Z., Wu, W.-T.: On relaxed greedy randomized Kaczmarz methods for solving large sparse linear systems. Appl. Math. Lett. 83, 21–26 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bai, Z.-Z., Wu, W.-T.: On greedy randomized coordinate descent methods for solving large linear least-squares problems. Numer. Linear Algebra Appl. 26, 1–15 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bai, Z.-Z., Wu, W.-T.: On partially randomized extended Kaczmarz method for solving large sparse overdetermined inconsistent linear systems. Linear Algebra Appl. 578, 225–250 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  7. Bai, Z.-Z., Wu, W.-T.: On greedy randomized augmented Kaczmarz method for solving large sparse inconsistent linear systems. SIAM J. Sci. Comput. 43, A3892–A3911 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  8. Bai, Z.-Z., Wang, L., Muratova, G.V.: On relaxed greedy randomized augmented Kaczmarz methods for solving large sparse inconsistent linear systems. East Asian J. Applied Math. 12, 323–332 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  9. Borkar, V., Karamchandani, N., Mirani, S.: Randomized Kaczmarz for rank aggregation from pairwise comparisons. In: IEEE Information Theory Workshop (ITW), Cambridge, 2016, pp 389–393 (2016)

  10. Carlton, M.: Probability and statistics for computer scientists. Am. Stat. 62, 271–272 (2008)

    Article  Google Scholar 

  11. Du, K.: Tight upper bounds for the convergence of the randomized extended Kaczmarz and Gauss-Seidel algorithms. Numer. Linear Algebra Appl., Article e2233 (2019)

  12. Du, K., Si, W., Sun, X.: Randomized extended average block Kaczmarz for solving least squares. SIAM J. Sci. Comput. 42, A3541–A3559 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  13. Du, Y., Hayami, K., Zheng, N., Morikuni, K., Yin, J.: Kaczmarz-type inner-iteration preconditioned flexible GMRES methods for consistent linear systems. SIAM J. Sci. Comput. 43, S345–S366 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  14. Gordon, R., Bender, R., Herman, G.: Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. J. Theor. Biol. 29, 471–481 (1970)

    Article  Google Scholar 

  15. Gower, R., Richtárik, P.: Stochastic dual ascent for solving linear systems, arXiv:1512.06890 (2015)

  16. Gower, R., Richtárik, P.: Randomized iterative methods for linear systems. SIAM J. Matrix Anal. Appl. 36, 1660–1690 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  17. Gu, C., Liu, Y.: Variant of greedy randomized Kaczmarz for ridge regression. Appl. Numer. Math. 143, 223–246 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  18. Guo, W., Chen, H., Geng, W., Lei, L.: A modified Kaczmarz algorithm for computerized tomographic image reconstruction. In: IEEE International Conference on Biomedical Engineering and Informatics, pp 1–4 (2009)

  19. Hadgu, A.: An application of ridge regression analysis in the study of syphilis data. Stat. Med. 3, 293–299 (1984)

    Article  Google Scholar 

  20. Hardy, H., Littlewood, J., Pslya, G.: Inequality. Bulletin of the American Mathematical Society, pp. 293–302 (1952)

  21. Hefny, A., Needell, D., Ramdas, A.: Rows versus columns: randomized Kaczmarz or Gauss-Seidel for ridge egression. SIAM J. Sci. Comput. 39, S528–S542 (2017)

    Article  MATH  Google Scholar 

  22. Jiang, Y., Wu, G., Jiang, L.: A Kaczmarz method with simple random sampling for solving large linear systems, arXiv:2011.14693 (2020)

  23. Kaczmarz, S.: Approximate solution of systems of linear equations. Int. J. Control. 35, 355–357 (1937)

    MathSciNet  Google Scholar 

  24. Lee, S., Kim, H.: Noise properties of reconstructed images in a kilo-voltage on-board imaging system with iterative reconstruction techniques: A phantom study. Phys. Med. 30, 365–373 (2014)

    Article  Google Scholar 

  25. Lei, Y., Zhou, D.: Learning theory of randomized sparse Kaczmarz method. SIAM J. Imaging Sci. 11, 547–574 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  26. Liu, J., Wright, S.: An accelerated randomized Kaczmarz algorithm. Math. Comput. 85, 153–178 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  27. Loera, J., Haddock, J., Needell, D.: A sampling Kaczmarz-Motzkin algorithm for linear feasibility. SIAM J. Sci. Comput. 39, S66–S87 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  28. Necoara, I.: Faster randomized block Kaczmarz algorithms. SIAM J. Matrix Anal. Appl. 40, 1425–1452 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  29. Ma, A., Needell, D., Ramdas, A.: Convergence properties of the randomized extended Gauss–Seidel and Kaczmarz methods. SIAM J. Matrix Anal. Appl. 36, 1590–1604 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  30. Needell, D., Srebro, W., Ward, R.: Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm. Math. Program. 155, 549–573 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  31. Needell, D., Tropp, J.: Paved with good intentions: Analysis of a randomized block Kaczmarz method. Linear Algebra Appl. 441, 199–221 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  32. Needell, D., Zhao, R., Zouzias, A.: Randomized block Kaczmarz method with projection for solving least squares. Linear Algebra Appl. 484, 322–343 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  33. Niu, Y., Zheng, B.: A greedy block Kaczmarz algorithm for solving large-scale linear systems. Appl. Math. Lett., 104, Article 106294 (2020)

  34. Nutini, J., Sepehry, B., Laradji, I., Schmidt, M., Koepke, H., Virani, A.: Convergence rates for greedy Kaczmarz algorithms, and faster randomized Kaczmarz rules using the orthogonality graph. arXiv:1612.07838 (2016)

  35. Ramlau, R., Rosensteiner, M.: An efficient solution to the atmospheric turbulence tomography problem using Kaczmarz iteration. Inverse Probl. 28, 095004 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  36. Steinerberger, S.: Randomized Kaczmarz converges along small singular vectors. IAM J. Matrix Anal. Appl. 42, 608–615 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  37. Steinerberger, S.: A weighted randomized Kaczmarz method for solving linear systems. Math. Comput. 90, 2815–2826 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  38. Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15, 262–278 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  39. Thoppe, G., Borkar, V., Manjunath, D.: A stochastic Kaczmarz algorithm for network tomography. Automatica 50, 910–914 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  40. van Lith, B., Hansen, P., Hochstenbach, M.: A twin error gauge for Kaczmarz’s iterations. SIAM J. Sci. Comput. 43, S173–S199 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  41. Wang, C., Ameya, A., Lu, Y.: Randomized Kaczmarz algorithm for inconsistent linear systems: An exact MSE analysis. In: International Conference on Sampling Theory and Applications, Washington DC, pp 498–502 (2015)

  42. Yang, X.: A geometric probability randomized Kaczmarz method for large scale linear systems. Appl. Numer. Math. 164, 139–160 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  43. Zhang, J.: A new greedy Kaczmarz algorithm for the solution of very large linear systems. Appl. Math. Lett. 91, 207–212 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  44. Zhang, Y., Li, H.: Greedy Motzkin-Kaczmarz methods for solving linear systems. Numer. Linear Algebra Appl. https://doi.org/10.1002/nla.2429 (2021)

  45. Zouzias, A., Freris, N.: Randomized extended Kaczmarz for solving least squares. SIAM J. Matrix Anal. Appl. 34, 773–793 (2013)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

Special thanks to the two anonymous reviewers for their insightful comments and invaluable suggestions that greatly improved the representation of this paper. Meanwhile, we are grateful to Prof. Zhong-Zhi Bai, Mr. Shunchang Li, and Dr. Bo Feng for helpful discussions on an early version of this paper.

Funding

Gang Wu is supported by the National Natural Science Foundation of China under grant 12271518, the Key Research and Development Project of Xuzhou Natural Science Foundation under grant KC22288, and the Open Project of Key Laboratory of Data Science and Intelligence Education of the Ministry of Education under grant DSIE202203.

Author information

Authors and Affiliations

  1. School of Mathematics, China University of Mining and Technology, Xuzhou, 221116, Jiangsu, People’s Republic of China

    Yutong Jiang, Gang Wu & Long Jiang

Authors
  1. Yutong Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Long Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gang Wu.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Communicated by: Lothar Reichel

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, Y., Wu, G. & Jiang, L. A semi-randomized Kaczmarz method with simple random sampling for large-scale linear systems. Adv Comput Math 49, 20 (2023). https://doi.org/10.1007/s10444-023-10018-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10444-023-10018-2

Keywords

Mathematics Subject Classification (2010)

Search

Navigation