Skip to main content
Springer Nature Link
Log in

Improving the stochastically controlled stochastic gradient method by the bandwidth-based stepsize

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

Stepsize plays an important role in the stochastic gradient method. The bandwidth-based stepsize allows us to adjust the stepsize within a banded region determined by some boundary functions. Based on the bandwidth-based stepsize, we propose a new method, namely SCSG-BD, for smooth non-convex finite-sum optimization problems. For the boundary functions 1/t, \(1/(t\log (t + 1))\) and \(1/t^p\) (\(p\in (0,1)\)), SCSG-BD converges sublinearly to a stationary point at a faster rate than the stochastically controlled stochastic gradient (SCSG) method under certain conditions. Moreover, SCSG-BD is able to converge linearly to the solution if the objective function satisfies the Polyak–Łojasiewicz condition. We also introduce the 1/t-Barzilai–Borwein stepsize for practical computation. Numerical experiments demonstrate that SCSG-BD performs better than SCSG and its variants.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability

The data that support the finding of this study are available from the corresponding author upon reasonable request.

Notes

  1. available on https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/,

References

  1. Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8(1), 141–148 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  2. Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010: 19th International Conference on Computational Statistics, pp. 177–186 (2010)

  3. Bottou, L., Curtis, F.E., Nocedal, J.: Optimization methods for large-scale machine learning. SIAM Rev. 60(2), 223–311 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  4. Dai, Y.H., Huang, Y.K., Liu, X.W.: A family of spectral gradient methods for optimization. Comput. Optim. Appl. 74, 43–65 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  5. Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM J. Optim. 23(4), 2341–2368 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  6. Gower, R.M., Loizou, N., Qian, X., et al.: SGD: General analysis and improved rates. In: International Conference on Machine Learning, pp. 5200–5209 (2019)

  7. Hastie, T., Tibshirani, R., Friedman, J.H., et al.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York (2009)

    Book  MATH  Google Scholar 

  8. Hazan, E., Kale, S.: Beyond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization. J. Mach. Learn. Res. 15(1), 2489–2512 (2014)

    MathSciNet  MATH  Google Scholar 

  9. Horváth, S., Lei, L.H., Richtárik, P., et al.: Adaptivity of stochastic gradient methods for nonconvex optimization. SIAM J. Math. Data Sci. 4(2), 634–648 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  10. Huang, Y.K., Dai, Y.H., Liu, X.W.: Equipping the Barzilai–Borwein method with the two dimensional quadratic termination property. SIAM J. Optim. 31(4), 3068–3096 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  11. Jordan, M.I., Mitchell, T.M.: Machine learning: trends, perspectives, and prospects. Science 349(6245), 255–260 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  12. Johnson, R., Zhang, T.: Accelerating stochastic gradient descent using predictive variance reduction. Adv. Neural Inf. Process. Syst. 26, 315–323 (2013)

    MATH  Google Scholar 

  13. Kasiviswanathan, S.P., Jin, H.: Efficient private empirical risk minimization for high-dimensional learning. In: International Conference on Machine Learning, pp. 488–497 (2016)

  14. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  MATH  Google Scholar 

  15. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  MATH  Google Scholar 

  16. Lei, L.H., Jordan, M.I.: Less than a single pass: stochastically controlled stochastic gradient method. In: Artificial Intelligence and Statistics, pp. 148–156 (2017)

  17. Lei, L.H., Jordan, M.I.: On the adaptivity of stochastic gradient-based optimization. SIAM J. Optim. 30(2), 1473–1500 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  18. Lei, L.H., Ju, C., Chen, J.B., et al.: Non-convex finite-sum optimization via SCSG methods. Adv. Neural Inf. Process. Syst. 30, 2345–2355 (2017)

    MATH  Google Scholar 

  19. Nguyen, L.M., Liu, J., Scheinberg, K., et al.: SARAH: a novel method for machine learning problems using stochastic recursive gradient. In: International Conference on Machine Learning, pp. 2613–2621 (2017)

  20. Nguyen, L.M., Nguyen, P.H., Richtárik, P., et al.: New convergence aspects of stochastic gradient algorithms. J. Mach. Learn. Res. 20(176), 1–49 (2019)

    MathSciNet  MATH  Google Scholar 

  21. Polyak, B.T.: Gradient methods for the minimisation of functionals. USSR Comput. Math. Math. Phys. 3(4), 864–878 (1963)

    Article  MATH  Google Scholar 

  22. Rakhlin, A., Shamir, O., Sridharan, K.: Making gradient descent optimal for strongly convex stochastic optimization. In: International Conference on Machine Learning, pp. 1571–1578 (2012)

  23. Reddi, S.J., Hefny, A., Sra, S., et al.: Stochastic variance reduction for nonconvex optimization. In: International Conference on Machine Learning, pp. 314–323 (2016)

  24. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22, 400–407 (1951)

    Article  MathSciNet  MATH  Google Scholar 

  25. Roux, N., Schmidt, M., Bach, F.: A stochastic gradient method with an exponential convergence rate for finite training sets. Adv. Neural Inf. Process. Syst. 25, 663–2671 (2012)

    MATH  Google Scholar 

  26. Sutskever, I., Martens, J., Dahl, G., et al.: On the importance of initialization and momentum in deep learning. In: International Conference on Machine Learning, pp. 1139–1147 (2013)

  27. Tan, C.H., Ma, S.Q., Dai, Y.H., et al.: Barzilai–Borwein step size for stochastic gradient descent. Adv. Neural Inf. Process. Syst. 29, 685–693 (2016)

    MATH  Google Scholar 

  28. Wang, X.Y., Magnússon, S., Johansson, M.: On the convergence of step decay step-size for stochastic optimization. Adv. Neural Inf. Process. Syst. 34, 14226–14238 (2021)

    MATH  Google Scholar 

  29. Wang, X.Y., Yuan, Y.X.: On the convergence of stochastic gradient descent with bandwidth-based step size. J. Mach. Learn. Res. 24(48), 1–49 (2023)

    MathSciNet  MATH  Google Scholar 

  30. Yu, T.T., Liu, X.W., Dai, Y.H., et al.: A minibatch proximal stochastic recursive gradient algorithm using a trust-region-like scheme and Barzilai–Borwein stepsizes. IEEE Trans. Neural Net. Learn. 32(10), 4627–4638 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  31. Yu, T.T., Liu, X.W., Dai, Y.H., et al.: Stochastic variance reduced gradient methods using a trust-region-like scheme. J. Sci. Comput. 87(1), 1–24 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  32. Yu, T.T., Liu, X.W., Dai, Y.H., et al.: Variable metric proximal stochastic variance reduced gradient methods for nonconvex nonsmooth optimization. J. Ind. Manag. Optim. 18(4), 2611–2631 (2022)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the associate editor and the anonymous referees for their valuable comments and suggestions.This work was supported by the National Natural Science Foundation of China (Grant No. 11701137) and Natural Science Foundation of Hebei Province (Grant No. A2021202010).

Author information

Authors and Affiliations

  1. School of Sciences, Hebei University of Technology, Tianjin, 300401, China

    Chenchen Liu & Yakui Huang

  2. Institute of Mathematics, Hebei University of Technology, Tianjin, 300401, China

    Yakui Huang

  3. School of Artificial Intelligence, Hebei University of Technology, Tianjin, 300401, China

    Dan Wang

Authors
  1. Chenchen Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yakui Huang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Dan Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Yakui Huang.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, C., Huang, Y. & Wang, D. Improving the stochastically controlled stochastic gradient method by the bandwidth-based stepsize. Comput Optim Appl 90, 941–968 (2025). https://doi.org/10.1007/s10589-025-00651-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-025-00651-6

Keywords

Mathematics Subject Classification