Abstract
In the last decade, the majority of new central processing units (CPU) have become multicore. To take advantage of these new architectures, we need programming languages that can express parallelisms. The programming language Golang is well known for providing developers with an easy programming model for communicating sequential process-style concurrency enabling programmers to easily write functions that will execute on the different cores of a modern multicore CPUs. Unfortunately, Golang does not support distributed execution of goroutines on clusters or distributed systems. In this paper, we extend the concurrency capabilities of Golang to a distributed cluster by providing a library called Gluster that is simple and easy to use. We developed a programming model that allows users to easily distribute work between machines, in a similar way as workloads are distributed in multicore CPUs by using operating system threads with libraries similar to Pthreads and OpenMP. Our Gluster solution is based on a single master node that connects to peers over a network and distributes work to these peers. The master node is able to send function arguments over the network to worker nodes as well as receive return values. Results using matrix multiplication show that our distributed implementation can speed up the goroutines by 5\(\times \) in a small 16 nodes cluster, but more importantly, it shows that the results are scalable to cluster size.
Similar content being viewed by others
References
GoLang (2018) Go -Lang reference. https://golang.org/doc/. Accessed 1 May 2018
Hoare CAR (1978) Communicating sequential processes. Commun. ACM 21(8):666–677. https://doi.org/10.1145/359576.359585
Go usage (2018) https://research.swtch.com/gophercount. Accessed 1 May 2018
Go Usage (2018) https://www.zdnet.com/article/googles-go-beats-java-c-python-to-programming-language-of-the-year-crown/. Accessed 5 May 2018
Jones S (2018) CUDA—new features and beyond, developer talk. GTC 2018—ID S8278
Prasertsang A, Pradubsuwun D (2016) Formal verification of concurrency in go. In: 2016 13th International Joint Conference on Computer Science and Software Engineering (JCSSE), Khon Kaen, pp 1–4
Togashi N, Klyuev V (2014) Concurrency in Go and Java: performance analysis. In: 2014 4th IEEE International Conference on Information Science and Technology, Shenzhen, pp 213–216
Ueda Y, Ohara M (2017) Performance competitiveness of a statically compiled language for server-side Web applications. In: 2017 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp 13–22
Lange J, Ng N, Toninho B, Yoshida N (2018) A static verification framework for message passing in Go using behavioural types. Accepted draft at ICSE 2018
Midtgaard J, Nielson F, Nielson H (2018) Process-local static analysis of synchronous processes. In: 25th Static Analysis Symposium
Jenkins L, Zhou T, Spear M (2017) Redesigning Gos built-in map to support concurrent operations. In: 2017 26th International Conference on Parallel Architectures and Compilation Techniques (PACT), Portland, OR, pp 14–26
Pasarella E, Vidal ME, Zoltan C (2016) Comparing Mapreduce and pipeline implementations for counting triangles. In: Proceedings XVI Jornadas sobre Programación y Lenguajes, PROLE 2016, Salamanca, Spain, 14–16th September 2016
Binet S (2018) Go-HEP: writing concurrent software with ease and Go. arXiV:1808.06529 https://go-hep.org
Togashi N, Klyuev V (2015) A novel approach for web development: a schedule management system using GAE/Go. In: 2015 IEEE 7th International Conference on Awareness Science and Technology (iCAST), Qinhuangdao, pp 55–59
Fang Z, Luo M, Anwar FM, Zhuang H, Gupta RK (2018) Go-realtime: a lightweight framework for multiprocessor real-time system in user space. SIGBED Rev 14(4):46–52
Scionti A, Mazumdar S (2017) Let’s go: a data-driven multi-threading support. In: Proceedings of the Computing Frontiers Conference (CF’17). ACM, New York, NY, USA, pp 287–290
Anurag VN (2018) Distributed computing with Go: practical concurrency and parallelism for Go applications. Packt Publishing, Birmingham
Proto-actor (2018) http://proto.actor/. Accessed 1 May 2018
Hewitt C, Bishop P, Steiger R (1973) A universal modular actor formalism for artificial intelligence. IJCAI
Lu C (2017) Gleam. https://github.com/chrislusf. Accessed 1 Jan 2018
Dean J, Ghemawat S (2004) MapReduce: simplified data processing on large clusters. In: OSDI04 Proceedings of the 6th Conference on Symposium on Operating Systems Design and Implementation, vol 6
OpenMPI forum website. http://www.mpi-forum.org/. Accessed 1 Feb 2018
Dagum I, Menon R (1998) OpenMP: an industry standard API for shared-memory programming. IEEE Comput Sci Eng 5(1):46–55
Lewis B, Berg D (1998) Multithreaded programming with Pthreads. Prentice-Hall, Inc., Upper Saddle River
Pike R (2011) Gobs of data. The Go blog. https://blog.golang.org/gobs-of-data. Accessed 24 Mar 2011
Algee-Hewitt M (2016) Counting words in HathiTrust with Python and MPI. Stanford Literary Lab, Stanford
Github Gluster. https://github.com/James-Whitney/gluster. Accessed 1 Oct 2018
Author information
Authors and Affiliations
Corresponding author
Additional information
James Whitney and Chandler Gifford have contributed equally to this work.
Appendices
Appendices
Appendix Gluster exposes the following functions to the user:
-
AddRunner(ip string)—Adds a runner with given ip on the default port.
-
AddRunnerPort(ip string, port int) Adds runner with specified port.
-
ImportFunctionFile(filename string) -imports .go file containing functions to be run on distributed machines.
-
RunDist(funct string, reply reflect.Type, args ...interface) int—runs a function on a runner and returns an id for the job.
-
JobDone(id int) bool—returns true if a job with the given id is finished and the result is set.
Rights and permissions
About this article
Cite this article
Whitney, J., Gifford, C. & Pantoja, M. Distributed execution of communicating sequential process-style concurrency: Golang case study. J Supercomput 75, 1396–1409 (2019). https://doi.org/10.1007/s11227-018-2649-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-018-2649-2