Abstract
We address the problem of minimizing objectives from the class of piecewise differentiable functions whose nonsmoothness can be encapsulated in the absolute value function. They possess local piecewise linear approximations with a discrepancy that can be bounded by a quadratic proximal term. This overestimating local model is continuous but generally nonconvex. It can be generated in its abs-normal form by a minor extension of standard algorithmic differentiation tools. Here we demonstrate how the local model can be minimized by a bundle-type method, which benefits from the availability of additional gray-box information via the abs-normal form. In the convex case our algorithm realizes the consistent steepest descent trajectory for which finite convergence was established earlier, specifically covering counterexamples where steepest descent with exact line-search famously fails. The analysis of the abs-normal representation and the design of the optimization algorithm are geared toward the general case, whereas the convergence proof so far covers only the convex case.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alt, W.: Numerische Verfahren der konvexen, nichtglatten Optimierung. Eine anwendungsorientierte Einführung. Teubner, Leipzig (2004)
Aubin, J.-P., Arriga, C.: Differential Inclusions. Set-Valued Maps and Viability Theory. Springer, Berlin (1984)
Bonnans, F., Gilbert, J.C., Lemaréchal, C., Sagastizábal, C.: Numerical Optimization. Theoretical and Practical Aspects. Transl. from the French. 2nd revised edn. Springer, Berlin (2006)
Clarke, F.: Optimization and Nonsmooth Analysis. SIAM, Philadelphia (1990)
Cominetti, R., Courdurier, M.: Coupling general penalty schemes for convex programming with the steepest descent and the proximal point algorithm. SIAM J. Optim. 13(3), 745–765 (2002)
de Oliveira, W., Sagastizábal, C.: Bundle methods in the XXIst century: a birds’-eye view. Pesquisa Operacional 34(3), 647–670 (2014)
Fiege, S., Griewank, A., Walther, A.: An exploratory line search for piecewise differentiable objective functions based on algorithmic differentiation. PAMM 12, 631–632 (2012)
Fourer, R.: A simplex algorithm for piecewise-linear programming. I. Derivation and proof. Math. Program. 33, 204–233 (1985)
Garey, M., Johnson, D.: Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman, San Francisco (1979)
Goffin, J.-L.: Subgradient optimization in nonsmooth optimization (including the soviet revolution). Doc. Math. Extra Vol., 277–290 (2012)
Griewank, A.: On stable piecewise linearization and generalized algorithmic differentiation. Opt. Meth. Softw. 28(6), 1139–1178 (2013)
Griewank, A., Bernt, J.-U., Randons, M., Streubel, T.: Solving Piecewise Linear Equations in abs-normal Form. Technical report, Humboldt Universität zu Berlin (2013). To appear in Linear Algebra and its Applications
Griewank, A., Walther, A.: Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. SIAM, Philadelphia (2008)
Fiege, S., Griewank, A., Kulshreshta, K., Walther, A.: An algorithm for nonsmooth optimization by successive piecewise linearization. Technical report, HU Berlin (2015)
Gürbüzbalaban, M., Overton, M.L.: On Nesterov’s nonsmooth Chebyshev–Rosenbrock functions. Nonlinear Anal.: Theory Methods Appl. 75(3), 1282–1289 (2012)
Hare, W., Nutini, J.: A derivative-free approximate gradient sampling algorithm for finite minimax problems. Comput. Optim. Appl. 56(1), 1–38 (2013)
Hiriart-Urruty, J.-B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms I. Springer, Berlin (1993)
Karmitsa, N., Mäkelä, M.: Limited memory bundle method for large bound constrained nonsmooth optimization: convergence analysis. Optim. Methods Softw. 25(6), 895–916 (2010)
Lemaréchal, C.: Nonsmooth Optimization and Descent Methods. Technical Report 78,4, IIASA (1978)
Lemaréchal, C., Sagastizábal, C.: Variable metric bundle methods: from conceptual to implementable forms. Math. Program. 76(3), 393–410 (1997)
Lewis, A., Overton, M.: Nonsmooth optimization via quasi-Newton methods. Math. Program. 141(1–2), 135–163 (2013)
Mifflin, R., Sagastizábal, C.: A science fiction story in nonsmooth optimization originating at IIASA. Doc. Math. Extra Vol., 291–300 (2012)
Nesterov, Y.: Lexicographic differentiation of nonsmooth functions. Math. Program. 104(2–3), 669–700 (2005)
Scholtes, S.: Introduction to Piecewise Differentiable Functions. Springer, Berlin (2012)
Shen, J., Han, L., Pang, J.S.: Switching and stability properties of conewise linear systems. ESAIM: Control Optim. Calc. Var. 16(3), 764–793 (2010)
Shor, N.Z.: Nondifferentiable Optimization and Polynomial Problems. Kluwer, Dordrecht (1998)
Walther, A., Griewank, A.: Combinatorial Scientific Computing. Chapter Getting Started with ADOL-C, pp. 181–202. Chapman-Hall CRC Computational Science (2012)
Wolfe, P.: A method of conjugate subgradients for minimizing nondifferentiable functions. Math. Program. Stud. 3, 145–173 (1975)
Yuan, G., Wei, Z., Wang, Z.: Gradient trust region algorithm with limited memory BFGS update for nonsmooth convex optimization. Comp. Opt. Appl. 54(1), 45–64 (2012)
Acknowledgments
We thank the anonymous referees for their valuable comments, which helped us to improve the quality of the paper. This material was based upon work supported in part by the U.S. Department of Energy, Office of Science, under Contract DE-AC02-06CH11357.
Author information
Authors and Affiliations
Corresponding author
Additional information
Government License Section: The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC02-06CH11357. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government.
Rights and permissions
About this article
Cite this article
Griewank, A., Walther, A., Fiege, S. et al. On Lipschitz optimization based on gray-box piecewise linearization. Math. Program. 158, 383–415 (2016). https://doi.org/10.1007/s10107-015-0934-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-015-0934-x