Skip to main content
SpringerLink
Log in

Toward Structured Parallel Programming: Send-Receive Considered Harmful

  • Chapter
  • First Online:
Models, Mindsets, Meta: The What, the How, and the Why Not?

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11200))

  • 462 Accesses

Abstract

During the software crisis of the 1960s, Dijkstra’s famous thesis “goto considered harmful” paved the way for structured programming. In this paper that is a modified version of the short communication [10], we suggest that many current difficulties of parallel programming based on message passing are caused by poorly structured communication, which is a consequence of using low-level send-receive primitives. We argue that, like goto in sequential programs, send-receive should be avoided as far as possible and replaced by collective operations in the setting of message passing. We dispute some widely held opinions about the apparent superiority of low-level, pairwise primitives over collective operations, and we present substantial theoretical and empirical evidence to the contrary in the context of MPI (Message Passing Interface).

We also briefly mention our recent results obtained in the broader context of programming for modern many-core parallel systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bernaschi, M., Iannello, G., Lauria, M.: Experimental results about MPI collective communication operations. In: Sloot, P., Bubak, M., Hoekstra, A., Hertzberger, B. (eds.) HPCN-Europe 1999. LNCS, vol. 1593, pp. 774–783. Springer, Heidelberg (1999). https://doi.org/10.1007/BFb0100638

    Chapter  Google Scholar 

  2. Bilardi, G., Herley, K., Pietracaprina, A., Pucci, G., Spirakis, P.: BSP vs. LogP. In: Eighth ACM Symposium on Parallel Algorithms and Architectures, pp. 25–32 (1996)

    Google Scholar 

  3. Böhm, C., Jacopini, G.: Flow diagrams, Turing machines and languages with only two formation rules. Commun. ACM 9, 366–371 (1966)

    Article  Google Scholar 

  4. Dahl, O.J., Dijkstra, E.W., Hoare, C.A.: Structured Programming. Academic Press, London (1975)

    MATH  Google Scholar 

  5. Dijkstra, E.W.: Go To statement considered harmful. Commun. ACM 11(3), 147–148 (1968)

    Article  MathSciNet  Google Scholar 

  6. Fischer, J., Gorlatch, S.: Turing universality of morphisms for parallel programming. In: Gorlatch, S., Lengauer, C. (eds.) Third Int. Workshop on Constructive Methods for Parallel Programming (CMPP 2002). Forschungsberichte der Fakultät IV - Elektrotechnik und Informatik, vol. 2002/07, pp. 81–98. Technische Universität Berlin, June 2002

    Google Scholar 

  7. van de Geijn, R.: Using PLAPACK: Parallel Linear Algebra Package. Scientific and Engineering Computation Series. MIT Press, Cambridge (1997)

    Google Scholar 

  8. Gorlatch, S.: Towards formally-based design of message passing programs. IEEE Trans. Softw. Eng. 26(3), 276–288 (2000). http://wwwmath.uni-muenster.de/pvs/publikationen/papers/GorTSE.ps.gz

    Article  Google Scholar 

  9. Gorlatch, S.: Send-recv considered harmful? myths and truths about parallel programming. In: Malyshkin, V. (ed.) PaCT 2001. LNCS, vol. 2127, pp. 243–257. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44743-1_24

    Chapter  MATH  Google Scholar 

  10. Gorlatch, S.: Send-receive considered harmful: myths and realities of message passing. ACM TOPLAS 26(1), 47–56 (2004)

    Article  MathSciNet  Google Scholar 

  11. Gorlatch, S.: Parallel skeletons. In: Padua, D. (ed.) Encyclopedia of Parallel Computing, pp. 1417–1422. Springer, Boston, MA (2011). https://doi.org/10.1007/978-0-387-09766-4_24

    Chapter  Google Scholar 

  12. Gorlatch, S., Wedler, C., Lengauer, C.: Optimization rules for programming with collective operations. In: Atallah, M. (ed.) Proceeding of the IPPS/SPDP 1999, pp. 492–499. IEEE Computer Society Press (1999)

    Google Scholar 

  13. Goudreau, M.W., Lang, K., Rao, S.B., Suel, T., Tsantilas, T.: Towards efficiency and portablility. Programming with the BSP model. In: Eighth ACM Symposium on Parallel Algorithms and Architectures, pp. 1–12 (1996)

    Google Scholar 

  14. Goudreau, M., Rao, S.: Single-message vs. batch communication. In: Heath, M., Ranade, A., Schreiber, R. (eds.) Algorithms for Parallel Processing, pp. 61–74. Springer, New York (1999)

    Chapter  Google Scholar 

  15. Gropp, W., Lusk, E., Skjellum, A.: Using MPI: Portable Parallel Programmingwith the Message Passing. MIT Press, Cambridge (1994)

    Google Scholar 

  16. Hagedorn, B., Steuwer, M., Gorlatch, S.: A transformation-based approach to developing high-performance GPU programs. In: Petrenko, A.K., Voronkov, A. (eds.) PSI 2017. LNCS, vol. 10742, pp. 179–195. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-74313-4_14

    Chapter  Google Scholar 

  17. Hagedorn, B., Stoltzfus, L., Steuwer, M., Gorlatch, S., Dubach, C.: High performance stencil code generation with Lift. In: Proceedings ACM CGO 2018, pp. 100–112 (2018). Best paper award

    Google Scholar 

  18. Hwang, K., Xu, Z.: Scalable Parallel Computing. McGraw Hill, New York (1998)

    MATH  Google Scholar 

  19. Kielmann, T., Bal, H.E., Gorlatch, S.: Bandwidth-efficient collective communication for clustered wide area systems. In: Parallel and Distributed Processing Symposium (IPDPS 2000), pp. 492–499 (2000)

    Google Scholar 

  20. Kielmann, T., Hofman, R.F., Bal, H.E., Plaat, A., Bhoedjang., R.A.: MagPIe: MPI’s collective communication operations for clustered wide area systems. In: Proceedings of the ACM PPoPP, pp. 131–140 (1999)

    Article  Google Scholar 

  21. Knoop, J., Steffen, B., Vollmer, J.: Parallelism for free: efficient and optimal bitvector analyses for parallel programs. ACM TOPLAS 18(3), 268–299 (1996)

    Article  Google Scholar 

  22. Kumar, V., et al.: Introduction to Parallel Computing. Benjamin/Cummings Publ, Redwood City (1994)

    Google Scholar 

  23. Pacheco, P.: Parallel Programming with MPI. Morgan Kaufmann Publ, San Francisco (1997)

    MATH  Google Scholar 

  24. Park, J.Y.L., Choi, H.A., Nupairoj, N., Ni, L.M.: Construction of optimal multicast trees based on the parameterized communication model. In: Proceedings of the International Conference on Parallel Processing (ICPP), vol. I, pp. 180–187 (1996)

    Google Scholar 

  25. Schneider, F.B.: On Concurrent Programming. Springer-Verlag, New York (1997). https://doi.org/10.1007/978-1-4612-1830-2

    Book  MATH  Google Scholar 

  26. Steuwer, M., Gorlatch, S.: Skelcl: A high-level extension of OpenCL formulti-GPU systems. J. Supercomput. 69(1), 25–33 (2014). https://doi.org/10.1007/s11227-014-1213-y

    Article  Google Scholar 

  27. Vadhiyar, S.S., Fagg, G.E., Dongarra, J.: Automatically tuned collective communications. In: Proceedings of the Supercomputing 2000. Dallas, TX, November 2000

    Google Scholar 

  28. Valiant, L.G.: General purpose parallel architectures. In: Handbook of Theoretical Computer Science, vol. A, Chap. 18, pp. 943–971. MIT Press (1990)

    Google Scholar 

Download references

Acknowledgements

I am grateful to many colleagues in the field of parallel computing, whose research provided necessary theoretical and experimental evidence to support the ideas presented here. It is my pleasure to acknowledge the very helpful comments of Chris Lengauer, Robert van de Geijn, Murray Cole, Jan Prins, Thilo Kielmann, Holger Bischof, and Phil Bacon on the preliminary version of the manuscript. The anonymous referees of [10] did a great job in improving the presentation.

Author information

Authors and Affiliations

  1. University of Muenster, Münster, Germany

    Sergei Gorlatch

Authors
  1. Sergei Gorlatch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sergei Gorlatch .

Editor information

Editors and Affiliations

  1. Lero–The Irish Software Research Center, University of Limerick, Limerick, Ireland

    Tiziana Margaria

  2. Verimag Laboratory, Grenoble, France

    Susanne Graf

  3. Aalborg University, Aalborg, Denmark

    Kim G. Larsen

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gorlatch, S. (2019). Toward Structured Parallel Programming: Send-Receive Considered Harmful. In: Margaria, T., Graf, S., Larsen, K. (eds) Models, Mindsets, Meta: The What, the How, and the Why Not?. Lecture Notes in Computer Science(), vol 11200. Springer, Cham. https://doi.org/10.1007/978-3-030-22348-9_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-22348-9_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-22347-2

  • Online ISBN: 978-3-030-22348-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation