Abstract
Nonlinear optimization algorithms are rarely discussed from a complexity point of view. Even the concept of solving nonlinear problems on digital computers is not well defined. The focus here is on a complexity approach for designing and analyzing algorithms for nonlinear optimization problems providing optimal solutions with prespecified accuracy in the solution space. We delineate the complexity status of convex problems over network constraints, dual of flow constraints, dual of multi-commodity, constraints defined by a submodular rank function (a generalized allocation problem), tree networks, diagonal dominant matrices, and nonlinear knapsack problem’s constraint. All these problems, except for the latter in integers, have polynomial time algorithms which may be viewed within a unifying framework of a proximity-scaling technique or a threshold technique. The complexity of many of these algorithms is furthermore best possible in that it matches lower bounds on the complexity of the respective problems.
In general nonseparable optimization problems are shown to be considerably more difficult than separable problems. We compare the complexity of continuous versus discrete nonlinear problems and list some major open problems in the area of nonlinear optimization.
Similar content being viewed by others
References
Ahuja, R. K., & Orlin, J. B. (2001b). Inverse optimization. Operations Research, 49, 771–783.
Ahuja, R. K., Batra, J. L., & Gupta, S. K. (1984). A parametric algorithm for the convex cost network flow and related problems. European Journal of Operational Research, 16, 222–235.
Ahuja, R. K., Hochbaum, D. S., & Orlin, J. B. (2003). Solving the convex cost integer dual network flow problem. Management Science, 49, 950–964.
Ahuja, R. K., Hochbaum, D. S., & Orlin, J. B. (2004). A cut based algorithm for the nonlinear dual of the minimum cost network flow problem. Algorithmica, 39, 189–208.
Ahuja, R. K., Magnanti, T. L., & Orlin, J. B. (1993). Network flows: Theory, algorithms and applications. New Jersey: Prentice Hall.
Baldick, R. (1991). A unification of polynomially solvable cases of integer ‘non-separable’ quadratic optimization. Lawrence Berkeley Laboratory manuscript.
Baldick, R., & Wu, F. F. (1990). Efficient integer optimization algorithms for optimal coordination of capacitors and regulators. IEEE Transactions on Power Systems, 5, 805–812.
Barahona, F. (1986). A solvable case of quadratic 0-1 programming. Discrete Applied Mathematics, 13, 23–28.
Barlow, R. E., Bartholomew, D. J., Bremer, J. M., & Brunk, H. D. (1972). Statistical inference under order restrictions. New York: Wiley.
Blum, M., Floyd, R. W., Pratt, V. R., Rivest, R. L., & Tarjan, R. E. (1972). Time bounds for selection. Journal of Computer Systems Science, 7, 448–461.
Brucker, P. (1984). An O(n) algorithm for quadratic knapsack problems. Operations Research Letters, 3, 163–166.
Burton, D., & Toint, Ph. L. (1992). On an instance of the inverse shortest paths problem. Mathematical Programming, 53, 45–61.
Burton, D., & Toint, Ph. L. (1994). On the use of an inverse shortest paths algorithm for recovering linearly correlated costs. Mathematical Programming, 63, 1–22.
Busacker, R. G., & Gowen, P. J. (1961) A procedure for determining minimal-cost network flow patterns. Operational Research Office, John Hopkins University, Baltimore, MD.
Cosares, S., & Hochbaum, D. S. (1994). A strongly polynomial algorithm for the quadratic transportation problem with fixed number of suppliers. Mathematics of Operations Research, 19, 94–111.
Dantzig, G. B. (1963). Linear programming and extensions. New Jersey: Princeton University Press.
Dennis, J. B. (1959). Mathematical programming and electrical networks. In Technology press research monographs (pp. 74–75). New York: Technology Press and Wiley.
Edmonds, J., & Karp, R. M. (1972). Theoretical improvements in algorithmic efficiency for network flow problems. Journal of ACM, 19, 248–264.
Erickson, R. E., Monma, C. L., & Veinott, A. F. (1987). Send-and-split method for minimum-concave-cost network flows. Mathematics of Operations Research, 12, 634–664.
Federgruen, A., & Groenevelt, H. (1986a). The greedy procedure for resource allocation problems: Necessary and sufficient conditions for optimality. Operations Research, 34, 909–918.
Federgruen, A., & Groenevelt, H. (1986b). Optimal flows in networks with multiple sources and sinks, with applications to oil and gas lease investment programs. Operations Research, 34, 218–225.
Frederickson, G. N., & Johnson, D. B. (1982). The complexity of selection and rankings in X+Y and matrices with sorted columns. Journal of Computing System Science, 24, 197–208.
Fourer, R. (1988). A simplex algorithm for piecewise-linear programming: Finiteness, feasibility and degeneracy. Mathematical Programming, 41, 281–316.
Gale, D. (1957). A theorem of flows in networks. Pacific Journal of Mathematics, 7, 1073–1082.
Gallo, G., Grigoriadis, M. D., & Tarjan, R. E. (1989). A fast parametric maximum flow algorithm and applications. SIAM Journal of Computing, 18, 30–55.
Garey, M., & Johnson, D. (1979). Computers and intractability, a guide to the theory of NP-completeness. New York: Freeman.
Goldberg, A. V., & Tarjan, R. E. (1988). A new approach to the maximum flow problem. Journal of the ACM, 35, 921–940.
Granot, F., & Skorin-Kapov, J. (1990). Some proximity and sensitivity results in quadratic integer programming. Mathematical Programming, 47, 259–268.
Guisewite, G., & Pardalos, P. M. (1990). Minimum concave cost network flow problems: Applications, complexity, and algorithms. Annals of Operations Research, 25, 75–100.
Hansen, P., & Simeone, B. (1986). Unimodular functions. Discrete Applied Mathematics, 14, 269–281.
Hochbaum, D. S. (1989). On a polynomial class of nonlinear optimization problems. Manuscript, U.C. Berkeley.
Hochbaum, D. S. (1993). Polynomial algorithms for convex network optimization. In D. Du, M. Pardalos (Eds.), Network optimization problems: algorithms, complexity and applications (pp. 63–92). Singapore: World Scientific.
Hochbaum, D. S. (1994). Lower and upper bounds for allocation problems. Mathematics of Operations Research, 19, 390–409.
Hochbaum, D. S. (1995). A nonlinear knapsack problem. Operations Research Letters, 17, 103–110.
Hochbaum, D. S. (1998). The pseudoflow algorithm for the maximum flow problem. Manuscript, UC Berkeley, revised 2003. Extended abstract in: Boyd & Rios-Mercado (Eds.), Lecture notes in computer science: Vol. 1412. The pseudoflow algorithm and the pseudoflow-based simplex for the maximum flow problem. Proceedings of IPCO98 (pp. 325–337), Bixby, June 1998. New York: Springer.
Hochbaum, D. S. (2001). An efficient algorithm for image segmentation, Markov random fields and related problems. Journal of the ACM, 48, 686–701.
Hochbaum, D. S. (2002). The inverse shortest paths problem. Manuscript, UC Berkeley.
Hochbaum, D. S. (2003). Efficient algorithms for the inverse spanning tree problem. Operations Research, 51, 785–797.
Hochbaum, D. S. (2005). Complexity and algorithms for convex network optimization and other nonlinear problems. 4OR, 3, 171–216.
Hochbaum, D. S., & Hong, S. P. (1995). About strongly polynomial time algorithms for quadratic optimization over submodular constraints. Mathematical Programming, 69, 269–309.
Hochbaum, D. S., & Hong, S. P. (1996). On the complexity of the production-transportation problem. SIAM Journal on Optimization, 6, 250–264.
Hochbaum, D. S., & Queyranne, M. (2003). The convex cost closure problem. SIAM Journal on Discrete Mathematics, 16, 192–207.
Hochbaum, D. S., & Seshadri, S. (1993). The empirical performance of a polynomial algorithm for constrained nonlinear optimization. Annals of Operations Research, 43, 229–248.
Hochbaum, D. S., & Shanthikumar, J. G. (1990). Convex separable optimization is not much harder than linear optimization. Journal of the ACM, 37, 843–862.
Hochbaum, D. S., Shamir, R., & Shanthikumar, J. G. (1992). A polynomial algorithm for an integer quadratic nonseparable transportation problem. Mathematical Programming, 55, 359–372.
Hoffman, A. J. (1960). Some recent applications of the theory of linear inequalities to extremal combinatorial analysis. In R. Bellman, M. Hall Jr. (Eds.), Proceedings of Symposia in Applied Mathematics: Vol. X. Combinatorial analysis (pp. 113–127). Providence: American mathematical Society.
Ibaraki, T., & Katoh, N. (1988). Resource allocation problems: Algorithmic approaches. Boston: MIT.
Iri, M. (1960). A new method of solving transportation network problems. Journal of the Operations Research Society of Japan, 3, 27–87.
Jewell, W. S. (1958). Optimal flow through networks. Technical report No. 8, Operations research Center, MIT, Cambridge.
Kapoor, S., & Vaidya, P. M. (1986). Fast algorithms for convex quadratic programming and multicommodity flows. In Proceedings of the 18th symposium on theory of computing (pp. 147–159).
Karzanov, A. V., & McCormick, S. T. (1997). Polynomial methods for separable convex optimization in unimodular linear spaces with applications. SIAM Journal on Computing, 26, 1245–1275.
Knuth, D. (1973). The art of computer programming: Vol. 3. Sorting and searching. Reading: Addison Wesley.
Kozlov, M. K., Tarasov, S. P., & Khachian, L. G. (1979). Polynomial solvability of convex quadratic programming. Doklady Akad. Nauk SSSR, 5, 1051–1053 (Translated in Soviet Mathematics Doklady 20 (1979), 1108–1111).
Lawler, E. (1979). Fast approximation algorithms for knapsack problems. Mathematics of Operations Research, 4, 339–356.
Mansour, Y., Schieber, B., & Tiwari, P. (1991). Lower bounds for computations with the floor operation. SIAM Journal on Computing, 20, 315–327.
Megiddo, N., & Tamir, A. (1993). Linear time algorithms for some separable quadratic programming problems. Operations Research Letters, 13, 203–211.
Minoux, M. (1984). A polynomial algorithm for minimum quadratic cost flow problems. European Journal of Operational Research, 18, 377–387.
Minoux, M. (1986). Solving integer minimum cost flows with separable convex cost objective polynomially. Mathematical Programming Study, 26, 237–239.
Minoux, M. (1986). Mathematical programming, theory and algorithms. Wiley: New York, Chaps. 5, 6.
Monteiro, R. D. C., & Adler (1989). Interior path following primal-dual algorithms. Part II: Convex quadratic programming. Mathematical Programming, 44, 43–66.
Moriguchi, S., & Shioura, A. (2004). On Hochbaum’s proximity-scaling algorithm for the general resource allocation problem. Mathematics of Operations Research, 29, 394–397.
Nemirovsky, A. S., & Yudin, D. B. (1983). Problem complexity and method efficiency in optimization. New York: Wiley.
Papadimitiou, C. H., & Steiglitz, K. (1982). Combinatorial optimization: algorithms and complexity. New Jersey: Prentice Hall.
Picard, J. C. (1976). Maximal closure of a graph and applications to combinatorial problems. Management Science, 22, 1268–1272.
Pinto, Y., & Shamir, R. (1994). Efficient algorithms for minimum-cost flow problems with piecewise-linear convex costs. Algorithmica, 11(3), 256–276.
Radzik, T. (1993). Parametric flows, Weighted means of cuts, and fractional combinatorial optimization. In P.M. Pardalos (Ed.), Complexity in numerical optimization (pp. 351–386). Singapore: World Scientific.
Renegar, J. (1987). On the worst case arithmetic complexity of approximation zeroes of polynomials. Journal of Complexity, 3, 90–113.
Rote, G., & Zachariasen, M. (2007, to appear). Matrix scaling by network flow. In Proceedings of SODA07.
Sahni, S. (1974). Computationally related Problems. SIAM Journal on Computing, 3, 262–279.
Shub, M., & Smale, S. (1996). Computational complexity: On the geometry of polynomials and a theory of cost, II. SIAM Journal on Computing, 15, 145–161.
Sun, J., Tsai, K. -H., & Qi, L. (1993). A simplex method for network programs with convex separable piecewise linear costs and its application to stochastic transshipment problems. In D. Du & P. M. Pardalos (Eds.), Network optimization problems: Algorithms, complexity and applications (pp. 283–300). Singapore: World Scientific.
Tamir, A. (1993). A strongly polynomial algorithm for minimum convex separable quadratic cost flow problems on series-parallel networks. Mathematical Programming, 59, 117–132.
Tardos, E. (1985). A strongly polynomial minimum cost circulation algorithm. Combinatorica, 5, 247–255.
Tardos, E. (1986). A strongly polynomial algorithm to solve combinatorial linear programs. Operations Research, 34, 250–256.
Värbrand, P., Tuy, H., Ghannadan, S., & Migdalas, A. (1995). The minimum concave cost network flow problems with fixed number of sources and non-linear arc costs. Journal of Global Optimisation, 6, 135–151.
Värbrand, P., Tuy, H., Ghannadan, S., & Migdalas, A. (1996). A strongly polynomial algorithm for a concave production-transportation problem with a fixed number of non-linear variables. Mathematical Programming, 72, 229–258.
Author information
Authors and Affiliations
Corresponding author
Additional information
An earlier version of this paper appeared in 4OR, 3:3, 171–216, 2005.
Rights and permissions
About this article
Cite this article
Hochbaum, D.S. Complexity and algorithms for nonlinear optimization problems. Ann Oper Res 153, 257–296 (2007). https://doi.org/10.1007/s10479-007-0172-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10479-007-0172-6