Skip to main content
SpringerLink
Log in

On the convergence of a class of inertial dynamical systems with Tikhonov regularization

  • Original Paper
  • Published:
Optimization Letters Aims and scope Submit manuscript

Abstract

We consider a class of inertial second order dynamical system with Tikhonov regularization, which can be applied to solving the minimization of a smooth convex function. Based on the appropriate choices of the parameters in the dynamical system, we first show that the function value along the trajectories converges to the optimal value, and prove that the convergence rate can be faster than \(o(1/t^2)\). Moreover, by constructing proper energy function, we prove that the trajectories strongly converges to a minimizer of the objective function of minimum norm. Finally, some numerical experiments have been conducted to illustrate the theoretical results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Attouch, H., Cominetti, R.: A dynamical approach to convex minimization coupling approximation with the steepest descent method. J. Differ. Equ. 128(2), 519–540 (1996)

    Article  MathSciNet  Google Scholar 

  2. Attouch, H., Czarnecki, M.O.: Asymptotic control and stabilization of nonlinear oscillators with non-isolated equilibria. J. Differ. Equ. 179(1), 278–310 (2002)

    Article  MathSciNet  Google Scholar 

  3. Attouch, H., Chbani, Z., Peypouquet, J., Redont, P.: Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity. Math. Program. 168(1–2), 123–175 (2018)

    Article  MathSciNet  Google Scholar 

  4. Attouch, H., Chbani, Z., Riahi, H.: Combining fast inertial dynamics for convex optimization with Tikhonov regularization. J. Math. Anal. Appl. 457(2), 1065–1094 (2018)

    Article  MathSciNet  Google Scholar 

  5. Attouch, H., Chbani, Z., Riahi, H.: Rate of convergence of the Nesterovs accelerated gradient method in the subcritical case \(\alpha \le 3\), ESAIM: control. Optim. Calculus Var. 25, 2 (2019)

    Article  Google Scholar 

  6. Attouch, H., Chbani, Z., Riahi, H.: Fast convex optimization via time scaling of damped inertial gradient dynamics, https://hal.archives-ouvertes.fr/hal-02138954

  7. Attouch, H., Chbani, Z., Riahi, H.: Fast proximal methods via time scaling of damped inertial dynamics. SIAM J. Optim. 29(3), 2227–2256 (2019)

    Article  MathSciNet  Google Scholar 

  8. Attouch, H., Peypouquet, J., Redont, P.: Fast convex minimization via inertial dynamics with Hessian driven damping. J. Differ. Equ. 261(10), 5734–5783 (2016)

    Article  Google Scholar 

  9. Bolte, J.: Continuous gradient projection method in Hilbert spaces. J. Optim. Theory Appli. 119(2), 235–259 (2003)

    Article  MathSciNet  Google Scholar 

  10. Bach, F.: Learning with submodular functions: a convex optimization perspective. Found. Trends Mach. Learn. 6(2–3), 145–373 (2013)

    Article  Google Scholar 

  11. Becker, S., Bobin, J., Cands, E.J.: NESTA: a fast and accurate first-order method for sparse recovery. SIAM J. Imag. Sci. 4(1), 1–39 (2011)

    Article  MathSciNet  Google Scholar 

  12. Bot, R.I., Csetnek, E.R.: Second order forward-backward dynamical systems for monotone inclusion problems. SIAM J. Control Optim. 54(3), 1423–1443 (2016)

    Article  MathSciNet  Google Scholar 

  13. Bot, R.I., Csetnek, E.R.: Approaching nonsmooth nonconvex optimization problems through first order dynamical systems with hidden acceleration and Hessian driven damping terms. Set-Valued Variation. Anal. 26(2), 227–245 (2018)

    Article  MathSciNet  Google Scholar 

  14. Bot, R.I., Csetnek, E.R.: A second-order dynamical system with Hessian-driven damping and penalty term associated to variational inequalities. Optimization 68(7), 1265–1277 (2019)

    Article  MathSciNet  Google Scholar 

  15. Bot, R.I., Csetnek, E.R., László, S.C.: Second-order dynamical systems with penalty terms associated to monotone inclusions. Anal. Appl. 16(05), 601–622 (2018)

    Article  MathSciNet  Google Scholar 

  16. Bot, R.I., Csetnek, E.R., László, S.C.: A second-order dynamical approach with variable damping to nonconvex smooth minimization. Appl. Anal. 99(3), 361–378 (2020)

    Article  MathSciNet  Google Scholar 

  17. Bot, R.I., Csetnek, E.R., László, S.C.: Tikhonov regularization of a second order dynamical system with Hessian driven damping. Math. Program. (2020). https://doi.org/10.1007/s10107-020-01528-8

    Article  Google Scholar 

  18. Bach, F., Jenatton, R., Mairal, J., Obozinski, G.: Structured sparsity through convex optimization. Stat. Sci. 27(4), 450–468 (2012)

    Article  MathSciNet  Google Scholar 

  19. Beck, A., Teboulle, M.: A fast iterative shrinkage-threshoiding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2(1), 183–202 (2009)

    Article  Google Scholar 

  20. Condat, L.: A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158(2), 460–479 (2013)

    Article  MathSciNet  Google Scholar 

  21. Csetnek, E.R.: Continuous dynamics related to monotone inclusions and non-smooth optimization problems. Set Valued Variation. Anal. (2020). https://doi.org/10.1007/s11228-020-00548-y

    Article  MathSciNet  MATH  Google Scholar 

  22. Haraux, A.: Systèmes dynamiques dissipatifs et applications, Rech. Math. Appl. 17, Masson, Paris. (1991)

  23. May, R.: On the strong convergence of the gradient projection algorithm with Tikhonov regularizing term, arXiv preprint arXiv:1910.07873, (2019)

  24. Muehlebach, M., Jordan, M. I.: A dynamical systems perspective on Nesterov acceleration, arXiv preprint arXiv:1905.07436, (2019)

  25. Mitchell, D., Ye, N., Sterck, H.D.: Nesterov acceleration of alternating least squares for canonical tensor decomposition. Numer. Linear Algbera Appl. (2020). https://doi.org/10.1002/nla.2297

    Article  MATH  Google Scholar 

  26. Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Soviet Math. Doklady 27(2), 372–376 (1983)

    MATH  Google Scholar 

  27. Sontag, E.D.: Mathematical Control Theory Deterministic Finite-Dimensional Systems. Texts in Applied Mathematics, vol. 6, 2nd edn. Springer, New York (1998)

    MATH  Google Scholar 

  28. Su, W., Boyd, S., Candés, E.J.: A differential equation for modeling Nesterovs accelerated gradient method: theory and insights. J. Mach. Learn. Res. 17, 1–43 (2016)

    MathSciNet  MATH  Google Scholar 

  29. Sutskever, I., Martens, J., Dahl, G., et al.: On the importance of initialization and momentum in deep learning. In: International Conference on Machine Learning, pp. 1139–1147 (2013)

  30. Sra, S., Nowozin, S., Wright, S.J.: Optimization for Machine Learning. Mit Press, Cambridge (2012)

    Google Scholar 

  31. Tseng, P.: On accelerated proximal gradient methods for convex-concave optimization, http://pages.cs.wisc.edu/~brecht/cs726docs/Tseng.APG.pdf (2008). Accessed 2019

  32. Wright, J., Ganesh, A., Rao, S.: Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. Advances in Neural Information Processing Systems 22, 2080–2088 (2009)

    Google Scholar 

  33. Wilson, A. C., Recht, B., Jordan, M. I.: A lyapunov analysis of momentum methods in optimization, arXiv preprint arXiv:1611.02635, (2016)

Download references

Acknowledgements

The authors would like to thank the editor and anonymous reviewers for their insight and helpful comments and suggestions which improve the quality of the paper. This work is also supported in part by NSFC 11801131, Natural Science Foundation of Hebei Province (Grant No. A2019202229), Science and Technology Project of Hebei Education Department (Grant No. QN2018101).

Author information

Authors and Affiliations

  1. School of Science, Hebei University of Technology, Tianjin, People’s Republic of China

    Bo Xu

  2. Institute of Mathematics, Hebei University of Technology, Tianjin, People’s Republic of China

    Bo Wen

Authors
  1. Bo Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bo Wen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, B., Wen, B. On the convergence of a class of inertial dynamical systems with Tikhonov regularization. Optim Lett 15, 2025–2052 (2021). https://doi.org/10.1007/s11590-020-01663-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-020-01663-3

Keywords

Search

Navigation