Abstract
It is little to find off-the-shelf research results in the current literature about how competent Volunteer Computing (VC) performs big data applications. This paper explores whether VC scales for a large number of volunteers when they commit churn and how large VC needs to scale in order to achieve the same performance as that by High Performance Computing (HPC) or computing grid for a given big data problem. To achieve the goal, this paper proposes a unification model to support the construction of virtual big data problems, virtual HPC clusters, computing grids or VC overlays on the same platform. The model is able to compare the competence of those computing facilities in terms of speedup vs number of computing nodes or volunteers for solving a big data problem. The evaluation results have demonstrated that all the computing facilities scale for the big data problem, with a computing grid or a VC overlay being in need of more or much more computing nodes or volunteers to achieve the same speedup as that of a HPC cluster. This paper has confirmed that VC is competent for big data problems as long as a large number of volunteers is available from the Internet.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anderson, D.P.: BOINC: a system for public-resource computing and storage. In: The Proceedings of 5th IEEE/ACM International Conference on Grid Computing, pp. 4–10 (2004)
ATLAS@home Project Status (2017). http://atlasathome.cern.ch/server_status.php
Climateprediction.net (2016). http://www.climateprediction.net
Costa, F., Silva, L., Dahlin, M.: Volunteer cloud computing: mapreduce over the internet. In: The Proceedings of IEEE International Symposium on Parallel and Distributed Processing Workshops and Ph.D Forum, pp. 1855–1862 (2011)
Costa, F., Silva, J.N., Veiga, L., Ferreira, P.: Large-scale volunteer computing over the internet. J. Internet Serv. Appl. 3(3), 329–346 (2012)
Costa, F., Veiga, L., Ferreira, P.: Internet-scale support for map-reduce processing. J. Internet Serv. Appl. 4, 18 (2013)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Dinh, D.: Hadoop Performance Evaluation, Ruprecht-Karls Universitat Heidelberg (2009). https://wr.informatik.uni-hamburg.de/_media/research/labs/2009/2009-12-tien_duc_dinh-evaluierung_von_hadoop-report.pdf
Donvito, G., Marzulli, G., Diacono, D.: Testing of several distributed file-systems (HDFS, Ceph and GlusterFS) for supporting the HEP experiments analysis. J. Phys. Conf. Ser. 513(4), 1–7 (2014)
DrugDiscovery@Home (2017). http://www.drugdiscoveryathome.com/
Fawkes Cluster (2017). https://www.usq.edu.au/research/support-development/development/eresearch/hpc/hardware
FiND@Home (2017). http://findah.ucd.ie
Fujin (2017). http://nci.org.au/systems-services/national-facility/peak-system/fujin/
Hadoop (2014). https://wiki.apache.org/hadoop/ProjectDescription
Isaac Newton Cluster (2017). https://my.cqu.edu.au/web/eresearch/hpc-systems
Korpela, E.J.: SETI@home, BOINC, and volunteer distributed computing. Ann. Rev. Earth Planet. Sci. 40, 69–87 (2012)
Lin, H., Ma, X., Archuleta, J., Feng, W.C., Gardner, M., Zhang, Z.: Moon: MapReduce on opportunistic environments. In: The Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, pp. 95–106 (2010)
Monsalve, S.A., Carballeira, F.G., Mateos, A.C.: Analyzing the performance of volunteer computing for data intensive applications. In: The Proceedings of International Conference on High Performance Computing and Simulation, pp. 597–604 (2016)
Alonso-Monsalve, S., GarcÃa-Carballeira, F., Calderón, A.: A new volunteer computing model for data-intensive applications. Concur. Comput. Pract. Exp. 24, e4198 (2017)
Australia National Broadband (2017). http://www.nbnco.com.au/learn-about-the-nbn/speed.html
Oracle: An Enterprise Architect’s Guide to Big Data - Reference Architecture Overview, Oracle Enterprise Architecture White Paper (2016)
Raijin (2017). http://nci.org.au/systems-services/national-facility/peak-system/raijin/
Ryden, M., Oh, K., Chandra, A., Weissman, J.: Nebula: distributed edge cloud for data intensive computing. In: The Proceedings of IEEE International Conference on Cloud Engineering, pp. 57–66 (2014)
Sarmenta, L.: Volunteer Computing, Ph.D thesis, Massachusetts Institute of Technology (2001)
SGI Altix XE Cluster (2017). http://www.itservices.qut.edu.au/researchteaching/hpc/hpc_infrastructure.jsp
Stoica, I., et al.: Chord: a scalable peer-to-peer lookup protocol for internet applications. IEEE/ACM Trans. Netw. 11(1), 17–32 (2003)
Tang, B., Moca, M., Chevalier, S., He, H., Fedak, G.: Towards mapreduce for desktop grid computing. In: The Proceedings of International Conference on P2P, Parallel, Grid, Cloud and Internet Computing, pp. 193–200 (2010)
TechPowerUp (2017). https://www.techpowerup.com/forums/threads/processor-gflops-compilation.94721/
Top500 (2017). https://www.top500.org/lists/2017/11/
Tinaroo (2017). https://rcc.uq.edu.au/tinaroo
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, W., Guo, W. (2018). The Competence of Volunteer Computing for MapReduce Big Data Applications. In: Zhou, Q., Gan, Y., Jing, W., Song, X., Wang, Y., Lu, Z. (eds) Data Science. ICPCSEE 2018. Communications in Computer and Information Science, vol 901. Springer, Singapore. https://doi.org/10.1007/978-981-13-2203-7_2
Download citation
DOI: https://doi.org/10.1007/978-981-13-2203-7_2
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-2202-0
Online ISBN: 978-981-13-2203-7
eBook Packages: Computer ScienceComputer Science (R0)