Abstract
The subject of this paper is inequality constrained black-box optimization with mesh adaptive direct search (MADS). The MADS search step can include additional strategies for accelerating the convergence and improving the accuracy of the solution. The strategy proposed in this paper involves building a quadratic model of the function and linear models of the constraints. The quadratic model is built by means of a second directional derivative-based Hessian update. The linear terms are obtained by linear regression. The resulting quadratic programming (QP) problem is solved with a dedicated solver and the original functions are evaluated at the QP solution. The proposed search strategy is computationally less expensive than the quadratically constrained QP strategy in the state of the art MADS implementation (NOMAD). The proposed MADS variant (QPMADS) and NOMAD are compared on four sets of test problems. QPMADS outperforms NOMAD on all four of them for all but the smallest computational budgets.
Similar content being viewed by others
References
Abramson, M.A., Audet, C.: Convergence of mesh adaptive direct search to second-order stationary points. SIAM J. Optim. 17(2), 606–619 (2006)
Abramson, M.A., Audet, C., Dennis Jr, J.E., Le Digabel, S.: OrthoMADS: a deterministic MADS instance with orthogonal directions. SIAM J. Optim. 20(2), 948–966 (2009)
Abramson, M.A., Frimannslund, L., Steihaug, T.: A subclass of generating set search with convergence to second-order stationary points. Optim. Methods Softw. 29(5), 900–918 (2014)
Andersen, M. S., Dahl, J., Vandenberghe, L.: CVXOPT, Release 1.1.6. Available at http://cvxopt.org/userguide/index.html (2014)
Audet, C., Dennis Jr, J.E.: Analysis of generalized pattern searches. SIAM J. Optim. 13(3), 889–903 (2002)
Audet, C., Dennis Jr, J.E.: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17(1), 188–217 (2006)
Audet, C., Dennis Jr, J.E.: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20(1), 445–472 (2009)
Audet, C., Ianni, A., Le Digabel, S., Tribes, C.: Reducing the number of function evaluations in mesh adaptive direct search algorithms. SIAM J. Optim. 24(2), 621–642 (2014)
Bűrmen, Á., Puhan, J., Tuma, T.: Grid restrained Nelder-Mead algorithm. Comput. Optim. Appl. 34(3), 359–375 (2006)
Clarke, F.: Optimization and Nonsmooth Analysis. SIAM, Philadelphia (1983)
Conn, A.R., Scheinberg, K., Vincente, L.N.: Introduction to Derivative-Free Optimization. SIAM, Philadelphia (2009)
Conn, A.R., Le Digabel, S.: Use of quadratic models with mesh-adaptive direct search for constrained black box optimization. Optim. Methods Softw. 28(1), 139–158 (2013)
Coope, I.D., Price, C.J.: Frame-based methods for unconstrained optimization. J. Optim. Theory Appl. 107(2), 261–274 (2000)
Custódio, A.L., Vincente, L.N.: Using sampling and simplex derivatives in patters search methods. SIAM J. Optim. 18(2), 537–555 (2007)
Custódio, A.L., Rocha, H., Vincente, L.N.: Incorporating minimum Frobenius norm models in direct search. Comput. Optim. Appl. 46(2), 265–278 (2010)
Davidon, W.C.: Variable metric method for minimization. SIAM J. Optim. 1(1), 1–17 (1991)
Frimannslund, L., Steihaug, T.: A generating set search method using curvature information. Comput. Optim. Appl. 38(1), 105–121 (2007)
Le Digabel, S.: Algorithm 909: NOMAD: nonlinear optimization with the MADS algorithm. ACM Trans. Math. Softw. 37(4), 44:1–44:15 (2011)
Leventhal, D., Lewis, A.S.: Randomized Hessian estimation and directional search. Optimization 60(3), 329–345 (2011)
Moré, J., Wild, S.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)
Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (2006)
Oliphant, T.E.: Python for scientific computing. Comput. Sci. Eng. 9(3), 10–20 (2007)
Rockafellar, R.: Generalized directional derivatives and subgradients of nonconvex functions. Can. J. Math. 32(2), 257–280 (1980)
Stewart, G.W.: The efficient generation of random orthogonal matrices with an application to condition estimators. SIAM. J. Numer. Anal. 17(3), 403–409 (1980)
Van Dyke, B., Asaki, T.J.: Using QR decomposition to obtain a new instance of mesh adaptive direct search with uniformly distributed polling directions. J. Optim. Theory Appl. 159(3), 805–821 (2013)
van Rossum, G., et al.: The Python Language Reference. Python Software Foundation, available at http://docs.python.org/ (2014)
Vincente, L., Custódio, A.: Analysis of direct searches for discontinuous functions. Math. Program. 133(1–2), 299–325 (2012)
Acknowledgments
The research was co-funded by the Ministry of Education, Science, and Sport (Ministrstvo za Šolstvo, Znanost in Šport) of the Republic of Slovenia through the Programme P2-0246 Algorithms and optimization methods in telecommunications. The authors would like to thank two anonymous reviewers and the associate editor for their comments which helped to improve the paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bűrmen, Á., Olenšek, J. & Tuma, T. Mesh adaptive direct search with second directional derivative-based Hessian update. Comput Optim Appl 62, 693–715 (2015). https://doi.org/10.1007/s10589-015-9753-5
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10589-015-9753-5
Keywords
- Black-box optimization
- Constrained optimization
- Mesh adaptive direct search
- Second directional derivative
- Hessian update
- Quadratic models
- Quadratic programming