Abstract
This paper investigates the behavior, both good and bad, of the Broyden–Fletcher–Goldfarb–Shanno algorithm for smooth minimization, when applied to nonsmooth functions. We consider three particular examples. We first present a simple nonsmooth example, illustrating how this variable metric method (in this case with an exact line search) typically succeeds despite nonsmoothness. We then study, computationally, the behavior of the method with an inexact line search on the same example and discuss the results. In further support of the heuristic effectiveness of the method despite nonsmoothness, we prove that, for the very simplest class of nonsmooth functions (maximums of two affine functions), the method cannot stall at a nonstationary point. On the other hand, we present a nonsmooth example where the method with an inexact line-search converges to a stationary point notwithstanding the presence of directions of linear descent. Finally, we briefly compare line-search and trust-region strategies for this method in the nonsmooth case.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
See the discussion of this point in [9, pp. 151–154].
References
Powell, M.J.D.: Some global convergence properties of a variable metric algorithm for minimization without exact line searches. In: Cottle, R., Lemke, R.W. (eds.) Nonlinear Programming. SIAM-AMS Proceedings 9, pp. 53–72. Providence (1976)
Powell, M.J.D.: Some properties of the variable metric algorithm. In: Lootsma, F.A. (ed.) Numerical Methods for Nonlinear Optimization, pp. 1–17. Academic Press, New York (1972)
Powell, M.J.D.: On the convergence of the variable metric algorithm. J. Inst. Math. Appl. 7, 21–36 (1971)
Li, D.H., Fukushima, M.: On the global convergence of the BFGS method for nonconvex unconstrained optimization problems. SIAM J. Optim. 11, 1054–1064 (2001)
Li, D.H., Fukushima, M.: A modified BFGS method and its global convergence in nonconvex minimization. J. Comput. Appl. Math. 129, 15–35 (2001)
Mascarenhas, W.F.: The BFGS method with exact line searches fails for non-convex objective functions. Math. Program. 99, 49–61 (2004)
Dai, Y.-H.: Convergence properties of the BFGS algorithm. SIAM J. Optim. 13, 693–701 (2003)
Lemaréchal, C.: Numerical experiments in nonsmooth optimization. In: Nurminski, E.A. (ed.) Progress in Nondifferentiable Optimization, pp. 61–84. IIASA, Laxenburg (1982)
Lewis, A.S., Overton, M.L.: Nonsmooth optimization via quasi-Newton methods. Math. Program. 141, 135–163 (2013)
Conn, A.R., Gould, N.I.M., Toint, P.L.: Trust Region Methods. MPS-SIAM Series on Optimization. SIAM, Philadelphia (2000)
Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (2006)
Lewis, A.S.: Active sets, nonsmoothness and sensitivity. SIAM J. Optim. 13, 702–725 (2003)
Gürbüzbalaban, M., Overton, M.L.: On Nesterov’s nonsmooth Chebyshev–Rosenbrock functions. Nonlinear Anal. Theory Methods Appl. 75, 1282–1289 (2012)
Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Springer, Berlin (1998)
Acknowledgments
The authors are grateful to Michael Overton for many helpful suggestions. The research was supported by National Science Foundation Grant DMS-0806057.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Asen L. Donchev.
Rights and permissions
About this article
Cite this article
Lewis, A.S., Zhang, S. Nonsmoothness and a Variable Metric Method. J Optim Theory Appl 165, 151–171 (2015). https://doi.org/10.1007/s10957-014-0622-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10957-014-0622-7
Keywords
- Broyden–Fletcher–Goldfarb–Shanno (BFGS)
- Nonsmooth
- Line search
- Partial smoothness
- Stationary point
- Trust region