Skip to main content
SpringerLink
Log in

Data parallel computing for network-structured optimization problems

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

Data level parallelism is a type of parallelism whereby operations are performed on many data elements concurrently, by many processors. These operations are (more or less) identical, and are executed in a synchronous, orderly fashion. This type of parallelism is used by massively parallel SIMD (i.e., Single Instruction, Multiple Data) architectures, like the Connection Machine CM-2, the AMT DAP and Masspar, and MIMD (i.e., Multiple Instruction, Multiple Data) architectures, like the Connection Machine CM-5. Data parallelism can also be described by a theoretical model of computation: the Vector-Random Access Machine (V-RAM).

In this paper we discuss practical approaches to the data-parallel solution of large scale optimization problems with network—or embedded-network—structures. The following issues are addressed: (1) The concept of dataparallelism, (2) algorithmic principles that lead to data-parallel decomposition of optimization problems with network—or embedded-network—structures, (3) specific algorithms for several network problems, (4) data-structures needed for efficient implementations of the algorithms, and (5) empirical results that highlight the performance of the algorithms on a data-parallel computer, the Connection Machine CM-2.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Amdahl, G., The validity of single processor approach to achieving large scale computing capabilities. InAFIPS Proceedings, volume 30, pages 483–485, 1967.

  2. Balas, E., Miller, D., Pekny, J., and Toth, P., A parallel shortest path algorithm for the assignment problem. Management science technical report msrr 552, Carnegie Mellon University, 1989.

  3. Benders, J.F., Partitioning procedures for solving mixed variables programming problems.Numerische Mathematik, 4:238–252, 1962.

    Google Scholar 

  4. Bertsekas, D.P., Distributed asynchronous relaxation methods for linear network flow problems. MIT report LIDS-P-1606, Nov. 1986.

  5. Bertsekas, D.P. and El Baz, D., Distributed asynchronous relaxation methods for convex network flow problems.SIAM Journal on Control and Optimization, 25(1):74–85, 1987.

    Google Scholar 

  6. Bertsekas, D.P. and Eckstein, J.: Dual coordinate step methods for linear network flow problems.Mathematical Programming, 42(2):203–243, 1988.

    Google Scholar 

  7. Bertsekas, D.P., Distributed relaxation methods for linear network flow problems. InProc. 25th IEEE Conference on Decision and Control, pages 2101–2106, 1986.

  8. Bertsekas, D.P., The auction algorithm: A distributed relaxation method for the assignment problem. InParallel Optimization on Novel Computer Architectures, Annals of Operations Research, volume 14, pages 105–124, 1988.

  9. Bertsekas, D.P., Auction algorithms for network flow problems: a tutorial introduction.Computational Optimization and Applications, 1(1):7–66, 1992.

    Google Scholar 

  10. Bertsekas, D.P., Hossein, P., and Tseng, P., Relaxation methods for network flow problems with convex arc costs.SIAM Journal on Control and Optimization, 25:1219–1243, 1987.

    Google Scholar 

  11. Bertsekas, D.P. and J.N. Tsitsiklis.Parallel and Distributed Computation: Numerical Methods. Prentice Hall, Englewood Cliffs, New Jersey, 1989.

    Google Scholar 

  12. Blelloch, G.E.,Vector Models for Data Parallel Computing. The MIT Press, Cambridge, Massachusetts, 1990.

    Google Scholar 

  13. Bregman, L.M., The relaxation method for finding the common point of convex sets and its application to the solution of problems in convex programming.USSR Computational Mathematics and Mathematical Physics, 7:200–217, 1967.

    Google Scholar 

  14. Censor, Y. and Lent, A., An iterative row-action method for interval convex programming.Journal of Optimization Theory and Applications, 34:321–353, 1981.

    Google Scholar 

  15. Censor, Y., De Pierro, A.R., Elfving, T., Herman, G.T., and Iusem, A.N., On iterative methods for linearly constrained entropy maximization. In A. Wakulicz, editor,Numerical Analysis and Mathematical Modelling, volume 24, pages 145–163. Banach Center Publications, PWN-Polish Scientific Publisher, Warsaw, Poland, 1990.

    Google Scholar 

  16. Censor, Y. and Segman, J., On block-iterative entropy maximization.Journal of Information and Optimization Sciences, 8:275–291, 1987.

    Google Scholar 

  17. Censor, Y. and Zenios, S. A., Interval-constrained matrix balancing.Linear Algebra and Its Applications, 150:393–421, 1991.

    Google Scholar 

  18. Censor, Y. and Zenios, S. A., On the use ofD-functions in primal-dual methods and the proximal minimization algorithm. In A. Ioffe, M. Marcus, and S. Reich, editors,Optimization and Nonlinear Analysis, pages 76–97, London, 1992. Pitman Research Notes in Mathematics, Series 244, Longman.

  19. Censor, Y. and Zenios, S.A., The proximal minimization algorithm withD-functions.Journal of Optimization Theory and Applications.73(3):455–468, 1992.

    Google Scholar 

  20. Censor, Y. and Zenios, S.A.,Introduction to methods of parallel optimization. Colóqui 190 Brasileiro de Matemática, Institute of Pure and Applied Mathematics, IMPA, Rio de Janeiro, Brazil, 1993.

  21. Chajakis, E.D. and Zenios, S.A., Synchronous and asynchronous implementations of relaxation algorithms for nonlinear network optimization.Parallel Computing, 17:873–894, 1991.

    Google Scholar 

  22. Christofides, N., An algorithm for the chromatic number of a graph.The Computer Journal, 14(1):38–39, 1971.

    Google Scholar 

  23. Dembo, R.S., Mulvey, J.M., and Zenios, S.A., Large scale nonlinear network models and their application.Operations Research, 37:353–372, 1989.

    Google Scholar 

  24. Eckstein, J.,Splitting Methods for Monotone Operators with Applications to Parallel Optimization.PhD thesis, Department of Civil Engineering, MIT, Cambridge, MA, 1989.

    Google Scholar 

  25. Eckstein, J., Implementing and running the alternating step method on the Connection Machine CM-2.ORSA Journal on Computing, 5(1):84–96, 1993.

    Google Scholar 

  26. Eckstein, J. and Bertsekas, D.P., On the Douglas-Rachford splitting method and proximal point algorithm for maximal monotone operators.Mathematical Programming, 55(3):293–318, 1992.

    Google Scholar 

  27. Eckstein, J., Qi, R., Ragulin, V.I., and Zenios, S.A., Data parallel implementations of dense linear programming algorithms. Report 92-05-06, Decision Sciences Department, The Wharton School, University of Pennsylvania, Philadelphia, PA 19104, 1992.

    Google Scholar 

  28. Flynn, M.J., Some computer organizations and their effectiveness.IEEE Transactions on Computers, C 21:948–960, 1972.

    Google Scholar 

  29. Gabay, D. and Mercier, B., A dual algorithm for the solution of nonlinear variational inequalities via finite element approximations.Computational Mathematics and Applications, 2:17–48, 1976.

    Google Scholar 

  30. Glowinski, R. and Marocco, A., Sur l'approximation par elements d'ordre un et al resolution par penalisationdualite d'une class de problemes de dirichlet non lineaires.Revue Francaise d'Automatique Informatique et Recherche Operationnelle, 9(R22):41–76, 1975.

    Google Scholar 

  31. Goldberg, A. and Tarjan, R.E., Solving minimum-cost flow problems by successive approximations.Mathematics of Operations Research, 15(3):430–466, 1990.

    Google Scholar 

  32. Goldberg, A.V.,Efficient Graph Algorithms for Sequential and Parallel Computers. PhD thesis, Electrical Engineering and Computer Science, Massachusetts Insitute of Technology, Cambridge, MA, February 1987.

    Google Scholar 

  33. Goldberg, A. V., and Tarjan, R.E., A new approach to the maximum flow problem. InProceedings 18th ACM STOC., pages 134–146, 1986.

  34. Habbal, M.B., Lerman, S.R., and Koutsopoulos, H.N., A decomposition algorithm for the all-pairs shortest path problem on massively parallel computer architectures. Working paper, Department of Civil Engineering, MIT, Cambridge, MA, 1991.

    Google Scholar 

  35. Hestenes, M.R., Multiplier and gradient methods.Journal of Optimization Theory and Applications, 4:303–320, 1969.

    Google Scholar 

  36. Journal of the American Academy of Arts and Sciences.Daedalus, Winter 1992.

  37. Kempa, D., Kennington, J., and Zaki, H., Performance characteristics of the Jacobi and Gauus-Seidel versions of the auction algorithm on the Alliant FX/8.ORSA Journal on Computing, 3(2):92–106,1991.

    Google Scholar 

  38. Kennington, J.L. and Wang, Z., An empirical analysis of the dense assignment problem: sequential and parallel implementations.ORSA Journal on Computing, 3(4):299–306, 1991.

    Google Scholar 

  39. Klincewicz, J.G., Implementing an exact Newton method for separable convex transportation problems.Networks, 19:95–105, 1989.

    Google Scholar 

  40. Li, X. and Zenios, S. A., Data level parallel solution of min-cost network flow problems using ε-relaxations.European Journal of Operational Research. (to appear).

  41. Li, X. and Zenios, S. A., A massively parallel ε-relaxation algorithm for linear transportation problems. In P.M. Pardalos, editor,Advances in Optimization and Parallel Computing, pages 164–176. Elsevier Science Publishers B.V., 1992.

  42. Lustig, I.J., Mulvey, J.M., and Carpenter, T.J., Formulating two-stage stochastic programs for interior point methods.Operations Research, 39:757–770, 1991.

    Google Scholar 

  43. Mangasarian, O.L., and Meyer, R.R., Nonlinear perturbations of linear programs.SIAM Journal on Control and Optimization, 17:745–752, 1979.

    Google Scholar 

  44. Martinet, B., Regularisation d'inequations variationelles par approximations successives.Revue Francaise d'Automatique Informatique et Recherche Operationnelle,4:154–159, 1970.

    Google Scholar 

  45. McKenna, M. and Zenios, S. A., An optimal parallel implementation of a quadratic transportation algorithm. InFourth SIAM Conference on Parallel Processing for Scientific Computing, pages 357–363, Philadelphia, PA, 1990. SIAM.

    Google Scholar 

  46. Mulvey, J.M. and Vladimirou, H., Evaluation of a parallel hedging algorithm for stochastic network programming. In R. Sharda, B.L. Golden, E. Wasil, O.Balci, and W.Stewart, editors,Impact of Recent Computer Advances on Operations Research. Pergamon Press, 1989.

  47. Mulvey, J.M. and Vladimirou, H., Stochastic network optimization models for investment planning.Annals of Operations Research, 20:187–217, 1989.

    Google Scholar 

  48. Murtagh, B.A. and Saunders, M.A., Minos user's guide. Report SOL 77-9, Department of Operations Research, Stanford University, California, 1977.

    Google Scholar 

  49. Nagurney, A., Kim, D-S., and Robinson, A.G., Serial and parallel equilibration of large-scale constrained matrix problems with application to the social and economic sciences.International Journal of Supercomputer Applications, 4:49–71, 1990.

    Google Scholar 

  50. Nielsen, S. and Zenios, S.A., Data structures for network algorithms on massively parallel architectures.Parallel Computing, 18:1033–1052, 1992.

    Google Scholar 

  51. Nielsen, S., and Zenios, S.A., Massively parallel algorithms for singly constrained convex programs.ORSA Journal on Computing, 4:166–181, 1992.

    Google Scholar 

  52. Nielsen, S. and Zenios, S.A., A massively parallel algorithm for nonlinear stochastic network problems.Operations Research, 41(2): 319–337, 1993.

    Google Scholar 

  53. Nielsen, S.S. and Zenios, S.A., Solving multistage stochastic network programs, Mathematical Programming, (to appear).

  54. Nielsen, S.S. and Zenios, S.A., Proximal minimizations withD-functions and the massively parallel solution of linear network programs.Computational Optimization and Applications, 1 (4):375–398, 1993.

    Google Scholar 

  55. Nielsen, S.S. and Zenios, S.A., Proximal minimizations withD-functions and the massively parallel solution of linear stochastic network programs.International Journal of Supercomputing Applications, 1994. (to appear).

  56. Phillips, C. and Zenios, S.A., Experiences with large scale network optimization on the Connection Machine. InThe Impact of Recent Computing Advances on Operations Research, volume 9, pages 169–180. Elsevier Science Publishing, 1989.

  57. De Pierro, A.R. and Lusem, A.N., A relaxed version of Bregman's method for convex programming.Journal of Optimization Theory and Applications, 5:421–440, 1986.

    Google Scholar 

  58. Powell, M.J.D., A method for nonlinear constraints in minimization problems. In R.Fletcher, editor,Optimization, pages 283–298. Academic Press, 1969.

  59. Qi, R-J. and Zenios, S.A., On the scalability of data parallel decomposition algorithms for stochastic programs.Journal of Parallel and Distributed Computing, 1994 (to appear).

  60. Rockafellar, R.T., Augmented Lagrangians and applications to proximal point algorithms in convex programming.Mathematics of Operations Research, 1: 97–116, 1976.

    Google Scholar 

  61. Rockafellar, R.T., Monotone operators and the proximal point algorithm.SIAM Journal on Control and Optimization, 14:877–898, 1976.

    Google Scholar 

  62. Rockafellar, R.T.,Network Flows and Monotropic Programming. John Wiley, N. York, 1984.

    Google Scholar 

  63. Rockafellar, R.T. and Wets, R.J.-B., Scenarios and policy aggregation in optimization under uncertainty.Mathematics of Operations Research, 16:119–147, 1991.

    Google Scholar 

  64. Ruszczynski, A., A regularized decomposition method for minimizing a sum of polyhedral functions.Mathematical Programming, 35:309–333, 1986.

    Google Scholar 

  65. Schneider, M.H., Matrix scaling, entropy minimization and conjugate duality. I. Existence conditions.Linear Algebra and Its Applications, 114/115:785–813, 1989.

    Google Scholar 

  66. Schneider, M.H. and Zenios, S.A., A comparative study of algorithms for matrix balancing.Operations Research, 38:439–455, 1990.

    Google Scholar 

  67. Van Slyke, R.M. and Wets, R.J., L-shaped linear programs with applications to optimal control and stochastic programming.SIAM Journal of Applied Mathematics, 17:638–663, 1969.

    Google Scholar 

  68. Storoy S. and Sorevik, T., An SIMD, fine-grained, parallel algorithm for the dense linear assignment problems. Working paper, Department of Informatics, University of Bergen, Bergen, Norway, 1992.

    Google Scholar 

  69. Tseng, P., Dual ascent methods for problems with strictly convex costs and linear constraints: A unified approach.SIAM Journal on Control and Optimization, 28:214–242, Jan. 1990.

    Google Scholar 

  70. Wein, J. and Zenios, S.A., On the massively parallel solution of the assignment problem.Journal of Parallel and Distributed Computing, 13:228–236, 1991.

    Google Scholar 

  71. Wets, R.J.B., Stochastic programs with fixed resources: the equivalent deterministic problem.SIAM Review, 16:309–339, 1974.

    Google Scholar 

  72. Zaki, H., A comparison of two algorithms for the assignment problem. Technical report OR-90-002, Department of Mechanical and Industrial Engineering, University of Illinois, Champaign-Urbana, 1990.

    Google Scholar 

  73. Zenios, S.A. and Lasken, R.A., Nonlinear network optimization on a massively parallel Connection Machine.Annals of Operations Research, 14:147–165, 1988.

    Google Scholar 

  74. Zenios, S.A. and Mulvey, J.M., Relaxation techniques for strictly convex network problems.Annals of Operations Research, 5:517–538, 1986.

    Google Scholar 

  75. Zenios, S.A., Matrix balancing on a massively parallel Connection Machine.ORSA Journal on Computing, 2:112–125, 1990.

    Google Scholar 

  76. Zenios, S.A., On the fine-grain decomposition of multicommodity transportation problems.SIAM Journal on Optimization, 1(4):643–669, 1991.

    Google Scholar 

  77. Zenios, S.A. and Censor, Y., Massively parallel row-action algorithms for some nonlinear transportation problems.SIAM Journal on Optimization, 1:373–400, 1991.

    Google Scholar 

  78. Zenios, S.A. and Lasken, R.A., The Connection Machines CM-1 and CM-2: Solving nonlinear network problems. InACM International Conference on Supercomputing, pages 648–658, St. Malo, France, 1988.

  79. Zenios, S.A., and Mulvey, J.M., A distributed algorithm for convex network optimization problems.Parallel Computing, 6:45–56, 1988.

    Google Scholar 

  80. Zenios, S.A., Qi, R., and Chajakis, E.D., A comparative study of parallel dual coordinate ascent implementations for nonlinear network optimization. In Coleman, T. and Li, Y., editors,Large Scale Numerical Optimization, pages 238–255. SIAM, 1990.

  81. Stavros, Zenios, A., and Zenios, A. Stefanos, Robust optimization for matrix balancing from noisy data. Report 92-01-02, Decision Sciences Department, the Wharton School, University of Pennsylvania, Philadelphia, PA 19104, 1992.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Decision Sciences Department, The Wharton School, University of Pennsylvania, 19104, Philadelphia, PA

    Stavros A. Zenios

  2. University of Cyprus, Nicosia, Cyprus

    Stavros A. Zenios

Authors
  1. Stavros A. Zenios
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zenios, S.A. Data parallel computing for network-structured optimization problems. Comput Optim Applic 3, 199–242 (1994). https://doi.org/10.1007/BF01299446

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF01299446

Keywords

Search

Navigation