Abstract
The increasing demand for big data processing leads to commercial off-the-shelf (COTS) and cloud-based big data analytics services. Giant cloud service vendors provide customized big data processing systems (BDPS), which are more cost-effective for operation and maintenance than self-owned platforms. End users can rent big data analytics services with a pay-as-you-go cost model. However, when users’ data size increases, they need to scale their rental BDPS in order to achieve approximately the same performance, such as task completion time and normalized system throughput. Unfortunately, there is no effective way to help end-users to choose between scale-up direction and scale-out direction to expand their existing rental BDPS. Moreover, there is no any metric to measure the scalability of BDPS, either. Furthermore, the performance of BDPS services at different time slots is not consistent due to co-location and workload placement policies in modern internet data centers. To this end, this paper proposes scalability metric for BDPS in clouds, which can mitigate the aforementioned issues. This scalability metric quantifies the scalability of BDPS consistently under different system expansion configurations. This paper also conducts experiments on real BDPS platforms and derives optimization approaches for better scalability of BDPS, such as file compression during Shuffle process in MapReduce. The experiment results demonstrate the validity of the proposed optimization strategies.























Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Alibaba Cloud E-MapReduce. https://www.alibabacloud.com/products/emapreduce/
Amazon EMR. https://aws.amazon.com/emr/
Apache HBase. https://hbase.apache.org/
Baidu BMR. https://cloud.baidu.com/product/bmr.html
BDAS Spark SQL. https://spark.apache.org/sql/
Cloudera Impala. https://www.cloudera.com/products/open-source/apache-hadoop/impala.html
CloudSuite. http://cloudsuite.ch/
Flink. https://flink.apache.org/
GZIP. http://www.gzip.org/
Hadoop. https://hadoop.apache.org/
Hive. https://hive.apache.org/
Microsoft Azure HDInsight. https://azure.microsoft.com/en-us/services/hdinsight/
OrangeFS. http://www.orangefs.org/
SNAPPY. http://google.github.io/snappy/
Spark. https://spark.csdn.net/
Ahmad AAS, Andras P (2018) Measuring the scalability of cloud-based software services. In: 2018 IEEE world congress on services (SERVICES). IEEE, pp 5–6. https://doi.org/10.1109/SERVICES.2018.00016
Ahmad F, Lee S, Thottethodi M, Vijaykumar T (2012) PUMA: purdue mapreduce benchmarks suite
Amdahl GM (1967) Validity of the single processor approach to achieving large scale computing capabilities. In: Proceedings of the April 18-20, 1967, spring joint computer conference. ACM, pp 483–485. https://doi.org/10.1145/1465482.1465560
Appuswamy R, Gkantsidis C, Narayanan D, Hodson O, Rowstron A (2013) Scale-up vs scale-out for Hadoop: time to rethink? In: Proceedings of the 4th annual symposium on cloud computing. ACM, p 20. https://doi.org/10.1145/2523616.2523629
Baru C, Bhandarkar M, Nambiar R, Poess M, Rabl T (2013) Benchmarking big data systems and the bigdata top100 list. Big Data 1(1):60–64. https://doi.org/10.1089/big.2013.1509
Chang BR, Tsai HF, Wang YA (2016) Optimized multiple platforms for big data analysis. In: 2016 IEEE second international conference on multimedia Big Data (BigMM). IEEE, pp 155–158. https://doi.org/10.1109/BigMM.2016.61
Chen Q, Zhang D, Guo M, Deng Q, Guo S (2010) Samr: a self-adaptive mapreduce scheduling algorithm in heterogeneous environment. In: 2010 10th IEEE international conference on computer and information technology. IEEE, pp 2736–2743. https://doi.org/10.1109/CIT.2010.458
Cooper BF, Silberstein A, Tam E, Ramakrishnan R, Sears R (2010) Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st ACM symposium on cloud computing. ACM, pp 143–154. https://doi.org/10.1145/1807128.1807152
Dede E, Fadika Z, Govindaraju M, Ramakrishnan L (2014) Benchmarking MapReduce implementations under different application scenarios. Future Gener Comput Syst 36:389–399. https://doi.org/10.1016/j.future.2014.01.001
Dharanipragada J, Padala S, Kammili B, Kumar V (2017) Tula: a disk latency aware balancing and block placement strategy for Hadoop. In: 2017 IEEE international conference on Big Data (Big Data). IEEE, pp 2853–2858. https://doi.org/10.1109/BigData.2017.8258253
Echihabi K, Zoumpatianos K, Palpanas T (2020) Big sequence management: on scalability. In: Proceedings of the IEEE international conference on Big Data. IEEE BigData
Elmubarak SA, Yousif A, Bashir MB (2017) Performance based ranking model for cloud SaaS services. Int J Inf Technol Comput Sci 9(1):65–71. https://doi.org/10.5815/ijitcs.2017.01.08
Ferdman M, Adileh A, Kocberber O, Volos S, Alisafaee M, Jevdjic D, Kaynak C, Popescu AD, Ailamaki A, Falsafi B (2012) Clearing the clouds: a study of emerging scale-out workloads on modern hardware. In: ACM SIGPLAN Notices, vol 47. ACM, pp 37–48. https://doi.org/10.1145/2150976.2150982
Gao J, Manjula K, Roopa P, Sumalatha E, Bai X, Tsai WT, Uehara T (2012) A cloud-based TaaS infrastructure with tools for SaaS validation, performance and scalability evaluation. In: 4th IEEE international conference on cloud computing technology and science proceedings. IEEE, pp 464–471. https://doi.org/10.1109/CloudCom.2012.6427555
Gao J, Pattabhiraman P, Bai X, Tsai WT (2011) SaaS performance and scalability evaluation in clouds. In: Proceedings of 2011 IEEE 6th international symposium on service oriented system (SOSE). IEEE, pp 61–71. https://doi.org/10.1109/SOSE.2011.6139093
Garate-Escamilla AK, El Hassani AH, Andres E (2019) Big data scalability based on spark machine learning libraries. In: Proceedings of the 2019 3rd international conference on Big Data research, pp 166–171
Garg N, Janakiram D (2018) Sparker: optimizing spark for heterogeneous clusters. In: 2018 IEEE international conference on cloud computing technology and science (CloudCom). IEEE, pp 1–8. https://doi.org/10.1109/CloudCom2018.2018.00017
Ghasemi E, Chow P (2016) Accelerating Apache Spark big data analysis with fpgas. In: 2016 Intl IEEE conferences on ubiquitous intelligence & computing, advanced and trusted computing, scalable computing and communications, cloud and Big Data computing, internet of people, and smart world congress (UIC/ATC/ScalCom/CBDCom/IoP/SmartWorld). IEEE, pp 737–744. https://doi.org/10.1109/UIC-ATC-ScalCom-CBDCom-IoP-SmartWorld.2016.0119
Ghazal A, Rabl T, Hu M, Raab F, Poess M, Crolotte A, Jacobsen HA (2013) BigBench: Towards an industry standard benchmark for big data analytics. In: Proceedings of the 2013 ACM SIGMOD international conference on Management of data. ACM, pp 1197–1208. https://doi.org/10.1145/2463676.2463712
Govindaraju V, Idicula S, Agrawal S, Vardarajan V, Raghavan A, Wen J, Balkesen C, Giannikis G, Agarwal N, Sedlar E (2017) Big data processing: scalability with extreme single-node performance. In: 2017 IEEE international congress on Big Data (BigData Congress). IEEE, pp 129–136. https://doi.org/10.1109/BigDataCongress.2017.26
Grama A, Gupta A, Kumar V (1996) Isoefficiency function: a scalability metric for parallel algorithms and architectures. IEEE Trans Parallel Distrib Syst 4(8):12–21
Grama AY, Gupta A, Kumar V (1993) Isoefficiency: measuring the scalability of parallel algorithms and architectures. IEEE Parallel Distrib Technol Syst Appl 1(3):12–21. https://doi.org/10.1109/88.242438
Gunther N, Puglia P, Tomasette K (2015) Hadoop superlinear scalability. Queue 13(5):20. https://doi.org/10.1145/2773212.2789974
Guo Y, Rao J, Cheng D, Zhou X (2016) Ishuffle: improving hadoop performance with shuffle-on-write. IEEE Trans Parallel Distrib Syst 28(6):1649–1662. https://doi.org/10.1109/TPDS.2016.2587645
Gustafson JL (1988) Reevaluating Amdahl’s law. Commun ACM 31(5):532–533. https://doi.org/10.1145/42411.42415
Henning S, Hasselbring W (2021) How to measure scalability of distributed stream processing engines? In: Companion of the ACM/SPEC international conference on performance engineering, pp 85–88
Huang S, Huang J, Dai J, Xie T, Huang B (2010) The HiBench benchmark suite: characterization of the MapReduce-based data analysis. In: 2010 IEEE 26th international conference on data engineering workshops (ICDEW 2010). IEEE, pp 41–51 (2010). https://doi.org/10.1109/ICDEW.2010.5452747
Hwang K, Bai X, Shi Y, Li M, Chen WG, Wu Y (2015) Cloud performance modeling with benchmark evaluation of elastic scaling strategies. IEEE Trans Parallel Distrib Syst 27(1):130–143. https://doi.org/10.1109/TPDS.2015.2398438
Hwang K, Shi Y, Bai X (2014) Scale-out vs. scale-up techniques for cloud performance and productivity. In: 2014 IEEE 6th international conference on cloud computing technology and science. IEEE, pp 763–768. https://doi.org/10.1109/CloudCom.2014.66
Iosup A, Epema D (2006) Grenchmark: a framework for analyzing, testing, and comparing grids. In: Sixth IEEE international symposium on cluster computing and the grid (CCGRID’06), vol 1. IEEE, pp 313–320. https://doi.org/10.1109/CCGRID.2006.49
Jiang C, Fan T, Gao H, Shi W, Liu L, Cerin C, Wan J (2020) Energy aware edge computing: a survey. Comput Commun 151:556–580
Jiang C, Fan T, Qiu Y, Wu H, Zhang J, Xiong N, Wan J (2018) Interdomain I/O optimization in virtualized sensor networks. Sensors 18(12):4395. https://doi.org/10.3390/s18124395
Jiang C, Han G, Lin J, Jia G, Shi W, Wan J (2019) Characteristics of co-allocated online services and batch jobs in internet data centers: a case study from Alibaba cloud. IEEE Access 7:22495–22508
Jiang C, Qiu Y, Shi W, Ge Z, Wang J, Chen S, Cerin C, Ren Z, Xu G, Lin J (2020) Characterizing co-located workloads in Alibaba cloud datacenters. IEEE Trans Cloud Comput
Jiang C, Wang Y, Ou D, Li Y, Zhang J, Wan J, Luo B, Shi W (2017) Energy efficiency comparison of hypervisors. Sustain Comput Inform Syst. https://doi.org/10.1016/j.suscom.2017.09.005
Jiang C, Wang Y, Ou D, Luo B, Shi W (2017) Energy proportional servers: where are we in 2016? In: 2017 IEEE 37th international conference on distributed computing systems (ICDCS). IEEE, pp 1649–1660. https://doi.org/10.1109/ICDCS.2017.285
Jiang C, Wang Y, Ou D, Qiu Y, Li Y, Wan J, Luo B, Shi W, Cerin C (2018) Ease: energy efficiency and proportionality aware virtual machine scheduling. In: 2018 30th international symposium on computer architecture and high performance computing (SBAC-PAD). IEEE, pp 65–68
Jogalekar P, Woodside M (2000) Evaluating the scalability of distributed systems. IEEE Trans Parallel Distrib Syst 11(6):589–603. https://doi.org/10.1109/71.862209
Kim K, Jeon K, Han H, Kim S.g, Jung H, Yeom HY (2008) Mrbench: a benchmark for MapReduce framework. In: 2008 14th IEEE international conference on parallel and distributed systems. IEEE, pp 11–18. https://doi.org/10.1109/ICPADS.2008.70
Lee JY, Lee JW, Kim SD, et al (2009) A quality model for evaluating software-as-a-service in cloud computing. In: 2009 seventh ACIS international conference on software engineering research, management and applications. IEEE, pp 261–266. https://doi.org/10.1109/SERA.2009.43
Li M, Tan J, Wang Y, Zhang L, Salapura V (2015) Sparkbench: a comprehensive benchmarking suite for in memory data analytic platform Spark. In: Proceedings of the 12th ACM international conference on computing frontiers. ACM, p 53. https://doi.org/10.1145/2742854.2747283
Li Z, Shen H (2017) Measuring scale-up and scale-out Hadoop with remote and local file systems and selecting the best platform. IEEE Trans Parallel Distrib Syst 28(11):3201–3214. https://doi.org/10.1109/TPDS.2017.2712635
Lin J (2018) Scale up or scale out for graph processing? IEEE Internet Comput 22(3):72–78. https://doi.org/10.1109/MIC.2018.032501520
Marco VS, Taylor B, Porter B, Wang Z (2017) Improving Spark application throughput via memory aware task co-location: a mixture of experts approach. In: Proceedings of the 18th ACM/IFIP/USENIX middleware conference. ACM, pp 95–108. https://doi.org/10.1145/3135974.3135984
Meena M, Bharadi VA (2016) Performance analysis of cloud based software as a service (SaaS) model on public and hybrid cloud. In: 2016 symposium on colossal data analysis and networking (CDAN). IEEE, pp 1–6. https://doi.org/10.1109/CDAN.2016.7570951
Meng H, Yu S, Liu F, Xiao N (2017) Research on memory management and cache replacement policies in Spark. Comput Sci 44(6):31–35. https://doi.org/10.11896/j.issn.1002-137X.2017.06.005
Ming Z, Luo, C, Gao W, Han R, Yang Q, Wang L, Zhan J (2013) BDGS: a scalable big data generator suite in big data benchmarking. In: Advancing Big Data benchmarks. Springer, pp 138–154. https://doi.org/10.1007/978-3-319-10596-3_11
Nguyen N, Khan MMH, Albayram Y, Wang K (2017) Understanding the influence of configuration settings: an execution model-driven framework for Apache Spark platform. In: 2017 IEEE 10th international conference on cloud computing (CLOUD). IEEE, pp 802–807. https://doi.org/10.1109/CLOUD.2017.119
Nguyen N, Khan MMH, Wang K (2016) Csminer: an automated tool for analyzing changes in configuration settings across multiple versions of large scale cloud software. In: 2016 IEEE 9th international conference on cloud computing (CLOUD). IEEE, pp 472–480. https://doi.org/10.1109/CLOUD.2016.0069
Ousterhout K, Rasti R, Ratnasamy S, Shenker S, Chun BG (2015) Making sense of performance in data analytics frameworks. In: 12th \(\{\)USENIX\(\}\) symposium on networked systems design and implementation (\(\{\)NSDI\(\}\) 15), pp 293–307
Pirzadeh P, Carey M, Westmann T (2017) A performance study of big data analytics platforms. In: 2017 IEEE international conference on Big Data (Big Data). IEEE, pp 2911–2920. https://doi.org/10.1109/BigData.2017.8258260
Qiu Y, Jiang C, Wang Y, Ou D, Li Y, Wan J (2019) Energy aware virtual machine scheduling in data centers. Energies 12(4):646. https://doi.org/10.3390/en12040646
Raïs I, Balouek-Thomert D, Orgerie A.C, Lefèvre L, Parashar M (2019) Leveraging energy-efficient non-lossy compression for data-intensive applications. In: 2019 international conference on high performance computing & simulation (HPCS). IEEE
Ruan X, Chen H (2017) Improving Shuffle I/O performance for big data processing using hybrid storage. In: 2017 international conference on computing, networking and communications (ICNC). IEEE, pp 476–480. https://doi.org/10.1109/ICCNC.2017.7876175
Sandel R, Shtern M, Fokaefs M, Litoiu M (2015) Evaluating cluster configurations for big data processing: an exploratory study. In: 2015 IEEE 9th international symposium on the maintenance and evolution of service-oriented and cloud-based environments (MESOCA). IEEE, pp 23–30. https://doi.org/10.1109/MESOCA.2015.7328122
Siegmund N, Grebhahn A, Apel S, Kästner C (2015) Performance-influence models for highly configurable systems. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering. ACM, pp 284–294. https://doi.org/10.1145/2786805.2786845
Sun X-H, Rover DT (1994) Scalability of parallel algorithm-machine combinations. IEEE Trans Parallel Distrib Syst 5(6):599–613. https://doi.org/10.1109/71.285606
Tsai WT, Huang Y, Shao Q (2011) Testing the scalability of SaaS applications. In: 2011 IEEE international conference on service-oriented computing and applications (SOCA). IEEE, pp 1–4. https://doi.org/10.1109/SOCA.2011.6166245
Wang G, Xu J, He B (2016) A novel method for tuning configuration parameters of Spark based on machine learning. In: 2016 IEEE 18th international conference on high performance computing and communications; IEEE 14th international conference on smart city; IEEE 2nd international conference on data science and systems (HPCC/SmartCity/DSS). IEEE, pp 586–593. https://doi.org/10.1109/HPCC-SmartCity-DSS.2016.0088
Wang K, Khan MMH (2015) Performance prediction for Apache Spark platform. In: 2015 IEEE 17th international conference on high performance computing and communications, 2015 IEEE 7th international symposium on cyberspace safety and security, and 2015 IEEE 12th international conference on embedded software and systems. IEEE, pp 166–173. https://doi.org/10.1109/HPCC-CSS-ICESS.2015.246
Wang K, Khan MMH, Nguyen N, Gokhale S (2016) Modeling interference for Apache Spark jobs. In: 2016 IEEE 9th international conference on cloud computing (CLOUD). IEEE, pp 423–431. https://doi.org/10.1109/CLOUD.2016.0063
Wang L, Zhan J, Gao W, Jiang Z, Ren R, He X, Luo C, Lu G, Li J (2018) BOPS, not FLOPS! A new metric and roofline performance model for datacenter computing. arXiv preprint arXiv:1801.09212
Wang L, Zhan J, Luo C, Zhu Y, Yang Q, He Y, Gao W, Jia Z, Shi Y, Zhang S, et al (2014) Bigdatabench: a big data benchmark suite from internet services. In: 2014 IEEE 20th international symposium on high performance computer architecture (HPCA). IEEE, pp 488–499. https://doi.org/10.1109/HPCA.2014.6835958
Xia Y, Yang F (2017/04) Locality-based partitioning for Spark. In: 2017 5th international conference on frontiers of manufacturing science and measuring technology (FMSMT 2017). Atlantis Press. https://doi.org/10.2991/fmsmt-17.2017.233
Xie R, Jia X (2015) Data transfer scheduling for maximizing throughput of big-data computing in cloud systems. IEEE Trans Cloud Comput 6(1):87–98. https://doi.org/10.1109/TCC.2015.2464808
Xu L, Li M, Zhang L, Butt AR, Wang Y, Hu ZZ (2016) MEMTUNE: dynamic memory management for in-memory data analytic platforms. In: 2016 IEEE international parallel and distributed processing symposium (IPDPS), pp 383–392. IEEE. https://doi.org/10.1109/IPDPS.2016.105
Yigitbasi N, Iosup A, Epema D, Ostermann S (2009) C-meter: a framework for performance analysis of computing clouds. In: 2009 9th IEEE/ACM international symposium on cluster computing and the grid. IEEE, pp 472–477. https://doi.org/10.1109/CCGRID.2009.40
Acknowledgements
This work is supported by the National Natural Science Foundation of China (No. 61972118), the Science and Technology Project of State Grid Corporation of China (Research and Application on Multi-Datacenters Cooperation & Intelligent Operation and Maintenance, No. 5700-202018194A-0-0-00),and the the Science and Technology Project of State Grid Corporation of China (No. SGSDXT00DKJS1900040).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Li, Y., Ou, D., Zhou, X. et al. Scalability and performance analysis of BDPS in clouds. Computing 104, 1425–1460 (2022). https://doi.org/10.1007/s00607-022-01056-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00607-022-01056-7