Skip to main content
SpringerLink
Log in

A dual gradient-projection method for large-scale strictly convex quadratic problems

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

The details of a solver for minimizing a strictly convex quadratic objective function subject to general linear constraints are presented. The method uses a gradient projection algorithm enhanced with subspace acceleration to solve the bound-constrained dual optimization problem. Such gradient projection methods are well-known, but are typically employed to solve the primal problem when only simple bound-constraints are present. The main contributions of this work are threefold. First, we address the challenges associated with solving the dual problem, which is usually a convex problem even when the primal problem is strictly convex. In particular, for the dual problem, one must efficiently compute directions of infinite descent when they exist, which is precisely when the primal formulation is infeasible. Second, we show how the linear algebra may be arranged to take computational advantage of sparsity that is often present in the second-derivative matrix, mostly by showing how sparse updates may be performed for algorithmic quantities. We consider the case that the second-derivative matrix is explicitly available and sparse, and the case when it is available implicitly via a limited memory BFGS representation. Third, we present the details of our Fortran 2003 software package DQP, which is part of the GALAHAD suite of optimization routines. Numerical tests are performed on quadratic programming problems from the combined CUTEst and Maros and Meszaros test sets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Note that the sign of the inner product \(\langle b, v \rangle \) is arbitrary, since, for \(-v\), \(M^T (-v) = 0\) and \(\langle b, -v \rangle < 0\). We shall refer to a negative Fredholm alternative as that for which the signs of the components of v are flipped.

  2. Available from http://galahad.rl.ac.uk/galahad-www/ along with a Matlab interface.

References

  1. Arioli, M., Laratta, A., Menchi, O.: Numerical computation of the projection of a point onto a polyhedron. J. Optim. Theory Appl. 43(4), 495–525 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  2. Axehill, D. Hansson, A.: A dual gradient projection quadratic programming algorithm tailored for model predictive control. In: Proceedings of the 47th IEEE Conference on Decision and Control, pp. 3057–3064, Cancun (2008)

  3. Bartlett, R.A., Biegler, L.T.: QPSchur: a dual, active-set, Schur-complement method for large-scale and structured convex quadratic programming. Optim. Eng. 7(1), 5–32 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  4. Benzi, M., Golub, G.H., Liesen, J.: Numerical solution of saddle point problems. Acta Numer. 14, 1–137 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  5. Betts, J.T., Frank, P.D.: A sparse nonlinear optimization algorithm. J. Optim. Theory Appl. 82, 519–541 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bierlaire, M., Toint, P.L., Tuyttens, D.: On iterative algorithms for linear least squares problems with bound constraints. Linear Algebra Appl. 143, 111–143 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  7. Bisschop, J., Meeraus, A.: Matrix augmentation and partitioning in the updating of the basis inverse. Math. Program. 13(3), 241–254 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  8. Björck, Å.: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996)

    Book  MATH  Google Scholar 

  9. Boggs, P.T., Tolle, J.W.: Sequential quadratic programming. Acta Numer. 4, 1–51 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  10. Boland, N.L.: A dual-active-set algorithm for positive semi-definite quadratic programming. Math. Program. Ser. A 78(1), 1–27 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  11. Byrd, R.H., Nocedal, J., Schnabel, R.B.: Representations of quasi-Newton matrices and their use in limited memory methods. Math. Program. 63(2), 129–156 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  12. Calamai, P.H., Moré, J.J.: Projected gradient methods for linearly constrained problems. Math. Program. 39(1), 93–116 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  13. Conn, A.R., Gould, N.I.M.: On the location of directions of infinite descent for nonlinear programming algorithms. SIAM J. Numer. Anal. 21(6), 302–325 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  14. Conn, A.R., Gould, N.I.M., Toint, P.L.: Global convergence of a class of trust region algorithms for optimization with simple bounds. SIAM J. Numer. Anal. 25(2), 433–460 (1988). See also same journal 26, 764–767 (1989)

  15. Conn, A.R., Gould, N.I.M., Toint, P.L.: Testing a class of methods for solving minimization problems with simple bounds on the variables. Math. Comput. 50, 399–430 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  16. Conn, A.R., Gould, N.I.M., Toint, P.L.: Trust-Region Methods. SIAM, Philadelphia (2000)

    Book  MATH  Google Scholar 

  17. Curtis, F.E., Han, Z.: Globally Convergent Primal-Dual Active-Set Methods with Inexact Subproblem Solves. Technical Report 14T-010, COR@L Laboratory, Department of ISE, Lehigh University, 2014. In second review for SIAM Journal on Optimization

  18. Curtis, F.E., Han, Z., Robinson, D.P.: A globally convergent primal-dual active-set framework for large-scale convex quadratic optimization. Comput. Optim. Appl. (2014). doi:10.1007/s10589-014-9681-9

  19. Curtis, F.E., Johnson, T.C., Robinson, D.P., Wachter, A.: An inexact sequential quadratic optimization algorithm for nonlinear optimization. SIAM J. Optim. 24(3), 1041–1074 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  20. Dominguez, J., González-Lima, M.D.: A primal-dual interior-point algorithm for quadratic programming. Numer. Algorithms 42(1), 1–30 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  21. Dorn, W.: Duality in quadratic programming. Q. Appl. Math. 18, 155–162 (1960)

    Article  MathSciNet  MATH  Google Scholar 

  22. Dostál, Z.: Optimal Quadratic Programming Algorithms: With Applications to Variational Inequalities. Springer Optimization and Its Applications, vol. 23. Springer, New York (2009)

    MATH  Google Scholar 

  23. Dostál, Z., Schöberl, J.: Minimizing quadratic functions subject to bound constraints with the rate of convergence and finite termination. Comput. Optim. Appl. 30(1), 23–43 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  24. Duff, I.S.: MA57—a code for the solution of sparse symmetric definite and indefinite systems. ACM Trans. Math. Softw. 30(2), 118–144 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  25. Duff, I.S., Reid, J.K.: The multifrontal solution of indefinite sparse symmetric linear equations. ACM Trans. Math. Softw. 9(3), 302–325 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  26. Fletcher, R.: An \(\ell _1\) penalty method for nonlinear constraints. In: Boggs, P.T., Byrd, R.H., Schnabel, R.B. (eds.) Numerical Optimization 1984, pp. 26–40. SIAM, Philadelphia (1985)

    Google Scholar 

  27. Forsgren, A., Gill, P.E., Wong, E.: Primal and dual active-set methods for convex quadratic programming. Math. Program. 159(1), 469–508 (2016)

  28. Friedlander, M.P., Gould, N.I.M., Leyffer, S., Munson, T.: A filter active-set trust-region method. Technical Report Preprint ANL/MCS-P1456-0907, Argonne National Laboratory, Illinois (2007)

  29. Friedlander, M.P., Leyffer, S.: Global and finite termination of a two-phase augmented Lagrangian filter method for general quadratic programs. SIAM J. Sci. Comput. 30(4), 1706–1729 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  30. Friedlander, M.P., Orban, D.: A primal-dual regularized interior-point method for convex quadratic programs. Math. Program. Comput. 4(1), 71–107 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  31. George, A., Liu, J.W.H.: Computer Solution of Large Sparse Positive Definite Systems. Prentice-Hall, Englewood Cliffs (1981)

    MATH  Google Scholar 

  32. Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: A Schur-complement method for sparse quadratic programming. In: Cox, M.G., Hammarling, S.J. (eds.) Reliable Scientific Computation, pp. 113–138. Oxford University Press, Oxford (1990)

    Google Scholar 

  33. Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: Inertia-controlling methods for general quadratic programming. SIAM Rev. 33(1), 1–36 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  34. Gill, P.E., Murray, W., Saunders, M.A.: User’s guide for QPOPT 1.0: a Fortran package for quadratic programming. Report SOL 95-4, Department of Operations Research, Stanford University, Stanford (1995)

  35. Gill, P.E., Murray, W., Saunders, M .A., Wright, M .H.: A Schur-complement method for sparse quadratic programming. In: Cox, M .G., Hammarling, S .J. (eds.) Reliable Numerical Computation, pp. 113–138. Oxford University Press, Oxford (1990)

    Google Scholar 

  36. Goldfarb, D., Idnani, A.: A numerically stable dual method for solving strictly convex quadratic programs. Math. Program. 27(1), 1–33 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  37. Gould, N.I.M., Orban, D., Robinson, D.P.: Trajectory-following methods for large-scale degenerate convex quadratic programming. Math. Program. Comput. 5(2), 113–142 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  38. Gould, N.I.M., Orban, D., Toint, P.L.: GALAHAD—a library of thread-safe fortran 90 packages for large-scale nonlinear optimization. ACM Trans. Math. Softw. 29(4), 353–372 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  39. Gould, N.I.M., Orban, D., Toint, P.L.: CUTEst: a constrained and unconstrained testing environment with safe threads for mathematical optimization. Comput. Optim. Appl. 60(3), 545–557 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  40. Gould, N.I.M., Toint, P.L.: A quadratic programming bibliography. Numerical Analysis Group Internal Report 2000-1, Rutherford Appleton Laboratory, Chilton, Oxfordshire, England, 2000. See http://www.numerical.rl.ac.uk/qp/qp.html

  41. Gould, N.I.M., Toint, P.L.: SQP methods for large-scale nonlinear programming. In: Powell, M.J.D., Scholtes, S. (eds.) System Modelling and Optimization. Methods, Theory and Applications, pp. 149–178. Kluwer Academic Publishers, Dordrecht (2000)

    Google Scholar 

  42. Gould, N.I.M., Toint, P.L.: An iterative working-set method for large-scale non-convex quadratic programming. Appl. Numer. Math. 43(1–2), 109–128 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  43. Gould, N.I.M., Toint, P.L.: Numerical methods for large-scale non-convex quadratic programming. In: Siddiqi, A.H., Kočvara, M. (eds.) Trends in Industrial and Applied Mathematics, pp. 149–179. Kluwer Academic Publishers, Dordrecht (2002)

    Chapter  Google Scholar 

  44. Gould, N.I.M., Loh, Y., Robinson, D.P.: A filter method with unified step computation for nonlinear optimization. SIAM J. Optim. 24(1), 175–209 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  45. Gould, N.I.M., Loh, Y., Robinson, D.P.: A filter SQP method: local convergence and numerical results. SIAM J. Optim. 25(3), 1885–1911 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  46. Gould, N.I.M., Robinson, D.P.: A second derivative SQP method: global convergence. SIAM J. Optim. 20(4), 2023–2048 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  47. Gould, N.I.M., Robinson, D.P.: A second derivative SQP method: local convergence and practical issues. SIAM J. Optim. 20(4), 2049–2079 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  48. Gould, N.I.M., Robinson, D.P.: A second derivative SQP method with a ‘trust-region-free’ predictor step. IMA J. Numer. Anal. 32(2), 580–601 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  49. Gu, Z., Rothberg, E., Bixby, R.: Gurobi Optimizer, version 5.5. 0. Software program (2013)

  50. Gupta, A.: WSMP: Watson Sparse Matrix Package Part I—Direct Solution of Symmetric Sparse System. Research Report RC 21886, IBM T. J. Watson Research Center, Yorktown Heights (2010)

  51. Hager, W.W., Hearn, D.W.: Application of the dual active set algorithm to quadratic network optimization. Comput. Optim. Appl. 1(4), 349–373 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  52. Hintermüller, M., Kunisch, K.: The primal-dual active set strategy as a semismooth Newton method. SIAM J. Optim. 13(3), 865–888 (2002). (electronic) (2003)

    Article  MathSciNet  MATH  Google Scholar 

  53. Hogg, J.D., Reid, J.K., Scott, J.A.: Design of a multicore sparse Cholesky factorization using DAGs. SIAM J. Sci. Comput. 32(6), 36273649 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  54. Hogg, J.D., Scott, J.A.: An indefinite sparse direct solver for large problems on multicore machines. Technical Report RAL-TR-2010-011, Rutherford Appleton Laboratory, Chilton (2010)

  55. Hogg, J.D., Scott, J.A.: HSL_MA97: a bit-compatible multifrontal code for sparse symmetric systems. Technical Report RAL-TR-2011-024, Rutherford Appleton Laboratory, Chilton (2011)

  56. ILOG CPLEX. High-performance software for mathematical programming and optimization (2005)

  57. Lancaster, P., Tismenetsky, M.: The Theory of Matrices: With Applications, 2nd edn. Academic Press, London (1985)

    MATH  Google Scholar 

  58. Lescrenier, M.: Convergence of trust region algorithms for optimization with bounds when strict complementarity does not hold. SIAM J. Numer. Anal. 28(2), 476–495 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  59. Maros, I., Meszaros, C.: A repository of convex quadratic programming problems. Optim. Methods Softw. 11–12, 671–681 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  60. Mészáros, C.: The BPMPD interior point solver for convex quadratic problems. Optim. Methods Softw. 11(1–4), 431–449 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  61. Morales, J.L., Nocedal, J., Wu, Y.: A sequential quadratic programming algorithm with an additional equality constrained phase. IMA J. Numer. Anal. 32, 553–579 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  62. Moré, J.J., Toraldo, G.: On the solution of large quadratic programming problems with bound constraints. SIAM J. Optim. 1(1), 93–113 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  63. Moré, J.J., Thuente, D.J.: Line search algorithms with guaranteed sufficient decrease. ACM Trans. Math. Softw. 20(3), 286–307 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  64. Moyh-ud Din, H., Robinson, D.P.: A solver for nonconvex bound-constrained quadratic optimization. SIAM J. Optim. 25(4), 2385–2407 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  65. Nocedal, J., Wright, S.J.: Numerical Optimization. Series in Operations Research, 2nd edn. Springer, Heidelberg (2006)

    MATH  Google Scholar 

  66. Polyak, R.A., Costa, J., Neyshabouri, S.: Dual fast projected gradient method for quadratic programming. Optim. Lett. (2012). doi:10.1007/s11590-012-0476-6

  67. Powell, M.J.D.: ZQPCVX a FORTRAN subroutine for convex quadratic programming. University, Department of Applied Mathematics and Theoretical Physics (1983)

  68. Reid, J.K., Scott, J.A.: An out-of-core sparse Cholesky solver. ACM Trans. Math. Softw. 36(2), 9 (2009)

    Article  MathSciNet  Google Scholar 

  69. Robinson, D.P., Feng, L., Nocedal, J., Pang, J.-S.: Subspace accelerated matrix splitting algorithms for asymmetric and symmetric linear complementarity problems. SIAM J. Optim. 23(3), 1371–1397 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  70. Schenk, O., Gärtner, K.: On fast factorization pivoting methods for symmetric indefinite systems. Electron. Trans. Numer. Anal. 23, 158–179 (2006)

    MathSciNet  MATH  Google Scholar 

  71. Schmid, C., Biegler, L.T.: Quadratic programming methods for reduced hessian sqp. Comput. Chem. Eng. 18(9), 817–832 (1994)

    Article  Google Scholar 

  72. Spellucci, P.: Solving general convex QP problems via an exact quadratic augmented Lagrangian with bound constraints. Techn. Hochsch, Fachbereich Mathematik (1993)

  73. Stefanov, S.M.: Polynomial algorithms for projecting a point onto a region defined by a linear constraint and box constraints in \(\mathbb{R}^n\). J. Appl. Math. 2004(5), 409–431 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  74. Vanderbei, R.J.: LOQO: an interior point code for quadratic programming. Optim. Methods Softw. 11(1–4), 451–484 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  75. Williams, J.W.J.: Algorithm 232. Heapsort. Commun. ACM 7, 347–348 (1964)

    Google Scholar 

  76. Yuan, G., Lu, S., Wei, Z.: A modified limited SQP method for constrained optimization. Appl. Math. 1(1), 8–17 (2010)

    Article  MATH  Google Scholar 

  77. Zhu, C., Rockafellar, R.T.: Primal-dual projected gradient algorithms for extended linear-quadratic programming. SIAM J. Optim. 3(4), 751–783 (1993)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors are very grateful to Iain Duff for providing extensions to MA57 to cope with both sparse forward solution and the Fredholm alternative, and to Jonathan Hogg and Jennifer Scott for doing the same for MA77 and MA97. We are also grateful to Iain, Jonathan, Jennifer, Mario Arioli and Tyrone Rees for helpful discussions on Fredholm issues. N.I.M. Gould’s research supported by EPSRC Grants EP/I013067/1 and EP/M025179/1.

Author information

Authors and Affiliations

  1. Scientific Computing Department, STFC-Rutherford Appleton Laboratory, Chilton, Oxfordshire, OX11 0QX, England, UK

    Nicholas I. M. Gould

  2. Department of Applied Mathematics and Statistics, The Johns Hopkins University, Baltimore, MD, 21218, USA

    Daniel P. Robinson

Authors
  1. Nicholas I. M. Gould
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel P. Robinson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel P. Robinson.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 335 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gould, N.I.M., Robinson, D.P. A dual gradient-projection method for large-scale strictly convex quadratic problems. Comput Optim Appl 67, 1–38 (2017). https://doi.org/10.1007/s10589-016-9886-1

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-016-9886-1

Keywords

Search

Navigation