Abstract
Approximately 60 years ago two seminal findings, the cutting plane and the subgradient methods, radically changed the landscape of mathematical programming. They provided, for the first time, the practical chance to optimize real functions of several variables characterized by kinks, namely by discontinuities in their derivatives. Convex functions, for which a superb body of theoretical research was growing in parallel, naturally became the main application field of choice. The aim of the paper is to give a concise survey of the key ideas underlying successive development of the area, which took the name of numerical nonsmooth optimization. The focus will be, in particular, on the research mainstreams generated under the impulse of the two initial discoveries.




Similar content being viewed by others
Notes
Given the features of the adopted machinery, we keep on denoting the current iterate (i.e., the estimate of a minimizer) by \(\overline{{\mathbf {x}}}_k\), although the methods involved in this class are not necessarily of the bundle type.
References
Akbari Z, Yousefpour R, Peyghami MR (2014) A new nonsmooth trust region algorithm for locally Lipschitz unconstrained optimization problems. J Optim Theory Appl 164:733–754
An LTH, Tao PD (2005) The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems. J Global Optim 133:23–46
Armijo L (1966) Minimization of functions having Lipschitz continuous first partial derivatives. Pac J Math 16:1–3
Astorino A, Miglionico G (2016) Optimizing sensor cover energy via DC programming. Optim Lett 10:355–368
Astorino A, Frangioni A, Gaudioso M, Gorgone E (2011) Piecewise quadratic approximations in convex numerical optimization. SIAM J Optim 21:1418–1438
Astorino A, Gaudioso M, Gorgone E (2017) A method for convex minimization based on translated first-order approximations. Numer Algorithms 76:745–760
Astorino A, Fuduli A, Gaudioso M (2019) A Lagrangian relaxation approach for binary multiple instance classification. IEEE Trans Neural Netw Learn Syst 30:2662–2671
Bagirov AM, Karasözen B, Sezer M (2008) Discrete gradient method: derivative-free method for nonsmooth optimization. J Optim Theory Appl 137:317–334
Bagirov AM, Karmitsa N, Mäkelä MM (2014) Introduction to nonsmooth optimization: theory, practice and software. Springer, Berlin
Bagirov AM, Gaudioso M, Karmitsa N, Mäkelä MM, Taheri S (eds) (2019) Numerical nonsmooth optimization—state of the art algorithms. Springer, Berlin (to appear)
Bahiense L, Maculan N, Sagastizábal C (2002) The volume algorithm revisited: relation with bundle methods. Math Program 94:41–69
Barahona F, Anbil R (2000) The volume algorithm: producing primal solutions with a subgradient method. Math Program 87:385–399
Barzilai J, Borwein JM (1988) Two-point step size gradient methods. IMA J Numer Anal 8:141–148
Beck A, Teboulle M (2003) Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper Res Lett 31:167–175
Ben-Tal A, Nemirovski A (2001) Lectures on modern optimization, MPS/SIAM series on optimization. SIAM, Philadelphia
Bertsekas DP (1995) Nonlinear programming. Athena Scientific, Belmont
Bertsekas DP (2009) Convex optimization theory. Athena Scientific, Belmont
Bertsekas DP, Mitter SK (1973) A descent numerical method for optimization problems with nondifferentiable cost functionals. SIAM J Control 11:637–652
Bertsimas D, Vempala S (2004) Solving convex programs by random walks. J ACM 51:540–556
Bonnans J, Gilbert J, Lemaréchal C, Sagastizábal C (1995) A family of variable metric proximal methods. Math Program 68:15–47
Brännlund U, Kiwiel KC, Lindberg PO (1995) A descent proximal level bundle method for convex nondifferentiable optimization. Oper Res Lett 17:121–126
Burke JV, Lewis AS, Overton ML (2005) A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J Optim 15:751–779
Burke JV, Lewis AS, Overton ML (2008) The speed of Shor’s R-algorithm. IMA J Numer Anal 28:711–720
Burke JV, Curtis FE, Lewis AS, Overton ML, Simões LEA (2019) Gradient sampling methods for nonsmooth optimization. In: Bagirov AM, Gaudioso M, Karmitsa N, Mäkelä M, Taheri S (eds) Numerical nonsmooth optimization—state of the art algorithms. Springer, Berlin (to appear)
Byrd RH, Nocedal J, Schnabel RB (1994) Representations of quasi-Newton matrices and their use in limited memory methods. Math Program 63:129–156
Chebyshëv PL (1961) Sur les questions de minima qui se rattachent a la représentation approximative des fonctions, 1859, in Oeuvres de P. L. Tchebychef, vol 1. Chelsea, New York, pp 273–378
Chen X, Fukushima M (1999) Proximal quasi-Newton methods for nondifferentiable convex optimization. Math Program 85:313–334
Cheney EW, Goldstein AA (1959) Newton’s method for convex programming and Tchebycheff approximation. Numer Math 1:253–268
Clarke FH (1983) Optimization and nonsmooth analysis. Wiley, Hoboken, pp 357–386
D’Antonio G, Frangioni A (2009) Convergence analysis of deflected conditional approximate subgradient methods. SIAM J Optim 20:357–386
de Ghellinck G, Vial J-P (1986) A polynomial Newton method for linear programming. Algorithmica 1:425–453
de Oliveira W (2019) Proximal bundle methods for nonsmooth DC programming. J Global Optim 75:523–563
de Oliveira W, Solodov M (2016) A doubly stabilized bundle method for nonsmooth convex optimization. Math Program 156:125–159
de Oliveira W, Sagastizábal C, Lemaréchal C (2014) Convex proximal bundle methods in depth: a unified analysis for inexact oracles. Math Program 148:241–277
Demyanov VF, Malozemov VN (1974) Introduction to minimax. Wiley, New York
Demyanov VF, Rubinov AM (1995) Constructive nonsmooth analysis. Verlag Peter Lang, Berlin
Demyanov AV, Demyanov VF, Malozemov VN (2002) Minmaxmin problems revisited. Optim Methods Softw 17:783–804
Demyanov AV, Fuduli A, Miglionico G (2007) A bundle modification strategy for convex minimization. Eur J Oper Res 180:38–47
Di Pillo G, Grippo L, Lucidi S (1993) A smooth method for the finite minimax problem. Math Program 60:187–214
Di Pillo G, Grippo L, Lucidi S (1997) Smooth transformation of the generalized minimax problem. J Optim Theory Appl 95:1–24
Dvurechensky PE, Gasnikov AV, Nurminski EA, Stonyakin FS (2019) Advances in low-memory subgradient optimization. In: Bagirov A, Gaudioso M, Karmitsa N, Mäkelä M, Taheri S (eds) Numerical nonsmooth optimization—state of the art algorithms. Springer, Berlin (to appear)
Elzinga J, Moore TG (1975) A central cutting plane algorithm for the convex programming problem. Math Program 8:134–145
Eremin II (1967) The method of penalties in convex programming. Dokl Acad Nauk USSR 173:748–751
Ermoliev YuM (1966) Methods of solution of nonlinear extremal problems. Cybernetics 2:1–16
Fasano G, Liuzzi G, Lucidi S, Rinaldi F (2014) A linesearch-based derivative-free approach for nonsmooth constrained optimization. SIAM J Optim 24:959–992
Fenchel W (1951) Convex cones, sets and functions. Lectures at Princeton University. Princeton University, Princeton
Frangioni A (1996) Solving semidefinite quadratic problems within nonsmooth optimization algorithms. Comput Oper Res 23:1099–1118
Frangioni A (2002) Generalized bundle methods. SIAM J Optim 13:117–156
Frangioni A (2019) Standard bundle methods: untrusted models and duality. In: Bagirov AM, Gaudioso M, Karmitsa N, Mäkelä M, Taheri S (eds) Numerical nonsmooth optimization—state of the art algorithms. Springer, Berlin (to appear)
Frangioni A, Gendron B, Gorgone E (2018) Dynamic smoothness parameter for fast gradient methods. Optim Lett 12:43–53
Fuduli A, Gaudioso M, Giallombardo G (2004) Minimizing nonconvex nonsmooth functions via cutting planes and proximity control. SIAM J Optim 14:743–756
Fuduli A, Gaudioso M, Giallombardo G, Miglionico G (2015) A partially inexact bundle method for convex semi-infinite minmax problems. Commun Nonlinear Sci Numer Simul 21:172–180
Fukushima M (1984) A descent algorithm for nonsmooth convex optimization. Math Program 30:163–175
Fukushima M, Qi L (1996) A globally and superlinearly convergent algorithm for nonsmooth convex minimization. SIAM J Optim 6:1106–1120
Gaudioso M (2019) A view of Lagrangian relaxation and its applications. In: Bagirov AM, Gaudioso M, Karmitsa N, Mäkelä M, Taheri S (eds) Numerical nonsmooth optimization—state of the art algorithms. Springer, Berlin (to appear)
Gaudioso M, Gorgone E (2010) Gradient set splitting in nonconvex nonsmooth numerical optimization. Optim Methods Softw 25:59–74
Gaudioso M, Monaco MF (1982) A bundle type approach to the unconstrained minimization of convex nonsmooth functions. Math Program 23:216–223
Gaudioso M, Monaco MF (1991) Quadratic approximations in convex nondifferentiable optimization. SIAM J Control Optim 29:1–10
Gaudioso M, Monaco MF (1992) Variants to the cutting plane approach for convex nondifferentiable optimization. Optimization 25:65–75
Gaudioso M, Giallombardo G, Miglionico G (2006) An incremental method for solving convex finite min–max problems. Math Oper Res 31:173–187
Gaudioso M, Giallombardo G, Miglionico G (2009) On solving the Lagrangian dual of integer programs via an incremental approach. Comput Optim Appl 44:117–138
Gaudioso M, Giallombardo G, Miglionico G (2018) Minimizing piecewise concave functions over polyhedra. Math Oper Res 43:580–597
Gaudioso M, Giallombardo G, Miglionico G, Bagirov AM (2018) Minimizing nonsmooth DC functions via successive DC piecewise-affine approximations. J Global Optim 71:37–55
Gaudioso M, Giallombardo G, Mukhametzhanov M (2018) Numerical infinitesimals in a variable metric method for convex nonsmooth optimization. Appl Math Comput 318:312–320
Gaudioso M, Giallombardo G, Miglionico G, Vocaturo E (2019) Classification in the multiple instance learning framework via spherical separation. Soft Comput. https://doi.org/10.1007/s00500-019-04255-1
Goffin J-L (1977) On convergence rates of subgradients optimization methods. Math Program 13:329–347
Goffin J-L, Haurie A, Vial J-P (1992) Decomposition and nondifferentiable optimization with the projective algorithm. Manag Sci 38:284–302
Goffin J-L, Gondzio J, Sarkissian R, Vial J-P (1997) Solving nonlinear multicommodity flow problems by the analytic center cutting plane method. Math Program 76B:131–154
Grippo L, Lampariello F, Lucidi S (1991) A class of nonmonotone stabilization methods in unconstrained optimization. Numer Math 59:779–805
Haarala N, Miettinen K, Mäkelä MM (2007) Globally convergent limited memory bundle method for large-scale nonsmooth optimization. Math Program 109:181–205
Hald J, Madsen K (1981) Combined LP and quasi-Newton methods for minimax optimization. Math Program 20:49–62
Hare W, Sagastizábal C (2010) A redistributed proximal bundle method for nonconvex optimization. SIAM J Optim 20:2242–2473
Helmberg C, Rendl F (2000) A spectral bundle method for semidefinite programming. SIAM J Optim 10:673–696
Hintermüller M (2001) A proximal bundle method based on approximate subgradients. Comput Optim Appl 20:245–266
Hiriart-Urruty J-B (1986) Generalized differentiability/duality and optimization for problems dealing with differences of convex functions. Lecture notes in economic and mathematical systems, vol 256. Springer, Berlin, pp 37–70
Hiriart-Urruty JB, Lemaréchal C (1993) Convex analysis and minimization algorithms, vol I and II. Springer, Berlin
Joki K, Bagirov AM, Karmitsa N, Mäkelä MM, Taheri S (2018) Double bundle method for finding clarke stationary points in nonsmooth DC programming. SIAM J Optim 28:1892–1919
Karmitsa N (2015) Diagonal bundle method for nonsmooth sparse optimization. J Optim Theory Appl 166:889–905
Kelley JE (1960) The cutting plane method for solving convex programs. J SIAM 8:703–712
Kiwiel KC (1983) An aggregate subgradient method for nonsmooth convex minimization. Math Program 27:320–341
Kiwiel KC (1985) Methods of descent for nondifferentiable optimization. Lecture notes in mathematics, vol 1133. Springer, Berlin
Kiwiel KC (1986) A method for solving certain quadratic programming problems arising in nonsmooth optimization. IMA J Numer Anal 6:137–152
Kiwiel KC (1990) Proximity control in bundle methods for convex nondifferentiable minimization. Math Program 46:105–122
Kiwiel KC (1994) A Cholesky dual method for proximal piecewise linear programming. Numer Math 68:325–340
Kiwiel KC (1996) Restricted step and Levenberg–Marquardt techniques in proximal bundle methods for nonconvex nondifferentiable optimization. SIAM J Optim 6:227–249
Kiwiel KC (1999) A bundle Bregman proximal method for convex nondifferentiable minimization. Math Program 85:241–258
Kiwiel KC (2004) Convergence of approximate and incremental subgradient methods for convex optimization. SIAM J Optim 14:807–840
Kiwiel KC (2006) A proximal bundle method with approximate subgradient linearizations. SIAM J Optim 16:1007–1023
Kiwiel KC (2007) Convergence of the gradient sampling algorithm for nonsmooth nonconvex optimization. SIAM J Optim 18:379–388
Kiwiel KC (2010) A nonderivative version of the gradient sampling algorithm for nonsmooth nonconvex optimization. SIAM J Optim 20:1983–1994
Lemaréchal C (1974) An algorithm for minimizing convex functions. In: Rosenfeld JL (ed) Proceedings IFIP ’74 congress, pp 20–25, North-Holland, Amsterdam
Lemaréchal C (1978) Nonsmooth optimization and descent methods. Report RR-78-4, IIASA, Laxenburg, Austria
Lemaréchal C (1975) An extension of Davidon methods to nondifferentiable problems. Math Program Study 3:95–109
Lemaréchal C (1981) A view of line-searches. In: Auslender A, Oettli W, Stoer J (eds) Optimization and optimal control. Lecture notes in control and information sciences, vol 30. Springer, Berlin
Lemaréchal C (1982) Numerical experiments in nonsmooth optimization. In: Nurminski EA (ed) Progress in nondifferentiable optimization CP-82-S8. IIASA, Laxenburg, pp 61–84
Lemaréchal C (1989) Nondifferentiable optimization. In: Nemhauser GL et al (eds) Handbooks in OR & MS, vol 1. North-Holland, Amsterdam
Lemaréchal C, Mifflin R (eds) (1978) Nonsmooth optimization. Pergamon Press, Oxford
Lemaréchal C, Sagastizábal C (1997) Variable metrics bundle methods: from conceptual to implementable forms. Math Program 76:393–410
Lemaréchal C, Strodiot J-J, Bihain A (1981) On a bundle algorithm for nonsmooth optimization. In: Mangasarian OL, Meyer RR, Robinson SM (eds) Nonlinear programming, vol 4. Academic Press, New York, pp 245–282
Lemaréchal C, Nemirovskii A, Nesterov Y (1995) New variants of bundle methods. Math Program 69:111–147
Levin AYu (1965) On an algorithm for minimization of convex functions. Sov Math Dokl 6:286–290
Levitin EC, Polyak BT (1966) Constrained minimization methods. J Comput Math Math Phys 6:787–823 (in Russian)
Luksǎn L, Vlček J (1998) A bundle-Newton method for nonsmooth unconstrained minimization. Math Program 83:373–391
Mäkelä MM (2002) Survey of bundle methods for nonsmooth optimization. Optim Methods Softw 17:1–29
Mäkelä MM, Neittaanmäki P (1992) Nonsmooth optimization. World Scientific, Singapore
Mifflin R (1982) A modification and an extension of Lemaréchal’s algorithm for nonsmooth minimization. Math Program Study 17:77–90
Mifflin R (1984) Stationarity and superlinear convergence of an algorithm for univariate locally Lipschitz constrained minimization. Math Program 28:50–71
Mifflin R (1996) A quasi-second order proximal bundle algorithm. Math Program 73:51–72
Mifflin R, Sagastizábal C (2005) A VU-algorithm for convex minimization. Math Program 104:583–608
Mifflin R, Sun D, Qi L (1998) Quasi-Newton bundle-type methods for nondifferentiable convex optimizations. SIAM J Optim 8:583–603
Mifflin R, Sagastizábal C (2012) A science fiction story in nonsmooth optimization originating at IIASA. Documenta mathematica extra volume: optimization stories, pp 291–300
Monaco MF (1987) An algorithm for the minimization of a convex quadratic function over a simplex. Technical report, Dipartimento di Sistemi, Universitá della Calabria, vol 56
Mordukhovich BS (2006) Variational analysis and generalized differentiation. Springer, Berlin
Moreau J-J (1965) Proximité et dualité dans un espace hilbertien. Bull Soc Math Fr 93:272–299
Nedić A, Bertsekas DP (2001) Incremental subgradient methods for nondifferentiable optimization. SIAM J Optim 12:109–138
Nemirovski A, Yudin D (1983) Problem complexity and method efficiency in optimization. Wiley, New York
Nesterov Yu (1995) Complexity estimates of some cutting plane methods based on the analytic barrier. Math Program 69:149–176
Nesterov Yu (2005) Smooth minimization of non-smooth functions. Math Program 103:127–152
Nesterov Yu (2009) Primal-dual subgradient methods for convex problems. Math Program 120:221–259
Nesterov Yu (2009) Universal gradient methods for convex optimization problems. Math Program 152:381–404
Noll D, Apkarian P (2005) Spectral bundle methods for non-convex maximum eigenvalue functions: first-order methods. Math Program 104:701–727
Nurminski EA (1982) Subgradient method for minimizing weakly convex functions and \(\epsilon \)-subgradient methods of convex optimization. In: Nurminski EA (ed) Progress in nondifferentiable optimization CP-82-S8. IIASA, Laxenburg, pp 97–123
Ouorou A (2009) A proximal cutting plane method using Chebychev center for nonsmooth convex optimization. Math Program 119:239–271
Polak E, Mayne DQ, Higgins JE (1991) Superlinearly convergent algorithm for min–max problems. J Optim Theory Appl 69:407–439
Polyak BT (1978) Subgradient methods: a survey of Soviet research. In: Lemaréchal C, Mifflin R (eds) Nonsmooth optimization. Pergamon Press, Oxford, pp 5–29
Polyak BT (1987) Introduction to optimization. Optimization Software Inc., New York
Pshenichnyi BN (1970) An algorithm for general problems of mathematical programming. Kybernetika 5:120–125 (in Russian)
Pshenichnyi BN (1978) Nonsmooth optimization and nonlinear programming. In: Lemaréchal C, Mifflin R (eds) Nonsmooth optimization. Pergamon Press, Oxford, pp 71–78
Pshenichnyi BN, Danilin YuM (1975) Numerical methods for extremum problems. Nauka, Moscow
Qi L, Sun J (1993) A nonsmooth version of Newton’s method. Math Program 58:353–368
Qi L, Sun J (1994) A trust region algorithm for minimization of locally Lipschitzian functions. Math Program 66:25–43
Rauf AI, Fukushima M (1998) Globally convergent BFGS method for nonsmooth convex optimization. J Optim Theory Appl 104:539–558
Rockafellar RT (1970) Convex analysis. Princeton University Press, Princeton
Rockafellar RT (1976) Monotone operators and the proximal point algorithm. SIAM J Control Optim 14:877–898
Schramm H, Zowe J (1992) A version of the bundle idea for minimizing a nonsmooth function: conceptual idea, convergence analysis, numerical results. SIAM J Optim 2:121–152
Shor NZ (1962) Application of the gradient method for the solution of network transportation problems. In: Notes, Scientific seminar on theory and application of cybernetics and operations research, academy of science, Kiev (in Russian)
Shor NZ (1985) Minimization methods for nondifferentiable functions. Springer, Berlin
Shor NZ (1998) Nondifferentiable optimization and polynomial problems. Kluwer Academic Publishers, Boston
Sonnevend G (1985) An analytic center for polyhedrons and new classes of global algorithms for linear (smooth, convex) programming. In: Prekopa A (ed) Lecture notes in control and information sciences, vol 84. Springer, New York, pp 866–876
Strekalovsky AS (1998) Global optimality conditions for nonconvex optimization. J Global Optim 12:415–434
Todd MJ (1986) The symmetric rank-one quasi-Newton algorithm is a space-dilation subgradient algorithm. Oper Res Lett 5:217–219
Tuy H (2016) Convex analysis and global optimization. Springer, Berlin
van Ackooij W, Sagastizábal C (2014) Constrained bundle methods for upper inexact oracles with application to joint chance constrained energy problems. SIAM J Optim 24:733–765
Vlček J, Luksǎn L (2001) Globally convergent variable metric method for nonconvex nondifferentiable unconstrained minimization. J Optim Theory Appl 111:407–430
Wolfe P (1975) A method of conjugate subgradients for minimizing nondifferentiable functions. Math Program Study 3:143–173
Wolfe P (1976) Finding the nearest point in a polytope. Math Program 11:128–149
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Gaudioso, M., Giallombardo, G. & Miglionico, G. Essentials of numerical nonsmooth optimization. 4OR-Q J Oper Res 18, 1–47 (2020). https://doi.org/10.1007/s10288-019-00425-x
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10288-019-00425-x