Skip to main content
SpringerLink
Log in

Scalability and performance analysis of BDPS in clouds

  • Regular Paper
  • Published:
Computing Aims and scope Submit manuscript

Abstract

The increasing demand for big data processing leads to commercial off-the-shelf (COTS) and cloud-based big data analytics services. Giant cloud service vendors provide customized big data processing systems (BDPS), which are more cost-effective for operation and maintenance than self-owned platforms. End users can rent big data analytics services with a pay-as-you-go cost model. However, when users’ data size increases, they need to scale their rental BDPS in order to achieve approximately the same performance, such as task completion time and normalized system throughput. Unfortunately, there is no effective way to help end-users to choose between scale-up direction and scale-out direction to expand their existing rental BDPS. Moreover, there is no any metric to measure the scalability of BDPS, either. Furthermore, the performance of BDPS services at different time slots is not consistent due to co-location and workload placement policies in modern internet data centers. To this end, this paper proposes scalability metric for BDPS in clouds, which can mitigate the aforementioned issues. This scalability metric quantifies the scalability of BDPS consistently under different system expansion configurations. This paper also conducts experiments on real BDPS platforms and derives optimization approaches for better scalability of BDPS, such as file compression during Shuffle process in MapReduce. The experiment results demonstrate the validity of the proposed optimization strategies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

References

  1. AdOC. http://www.labri.fr/perso/ejeannot/old/adoc/adoc.html

  2. Alibaba Cloud E-MapReduce. https://www.alibabacloud.com/products/emapreduce/

  3. Amazon EMR. https://aws.amazon.com/emr/

  4. Apache HBase. https://hbase.apache.org/

  5. Baidu BMR. https://cloud.baidu.com/product/bmr.html

  6. BDAS Spark SQL. https://spark.apache.org/sql/

  7. BZIP2. https://www.sourceware.org/bzip2/

  8. Cloudera Impala. https://www.cloudera.com/products/open-source/apache-hadoop/impala.html

  9. CloudSuite. http://cloudsuite.ch/

  10. Flink. https://flink.apache.org/

  11. GZIP. http://www.gzip.org/

  12. Hadoop. https://hadoop.apache.org/

  13. Hive. https://hive.apache.org/

  14. LZO. http://www.oberhumer.com/opensource/lzo/

  15. Microsoft Azure HDInsight. https://azure.microsoft.com/en-us/services/hdinsight/

  16. OrangeFS. http://www.orangefs.org/

  17. SNAPPY. http://google.github.io/snappy/

  18. Spark. https://spark.csdn.net/

  19. Ahmad AAS, Andras P (2018) Measuring the scalability of cloud-based software services. In: 2018 IEEE world congress on services (SERVICES). IEEE, pp 5–6. https://doi.org/10.1109/SERVICES.2018.00016

  20. Ahmad F, Lee S, Thottethodi M, Vijaykumar T (2012) PUMA: purdue mapreduce benchmarks suite

  21. Amdahl GM (1967) Validity of the single processor approach to achieving large scale computing capabilities. In: Proceedings of the April 18-20, 1967, spring joint computer conference. ACM, pp 483–485. https://doi.org/10.1145/1465482.1465560

  22. Appuswamy R, Gkantsidis C, Narayanan D, Hodson O, Rowstron A (2013) Scale-up vs scale-out for Hadoop: time to rethink? In: Proceedings of the 4th annual symposium on cloud computing. ACM, p 20. https://doi.org/10.1145/2523616.2523629

  23. Baru C, Bhandarkar M, Nambiar R, Poess M, Rabl T (2013) Benchmarking big data systems and the bigdata top100 list. Big Data 1(1):60–64. https://doi.org/10.1089/big.2013.1509

    Article  Google Scholar 

  24. Chang BR, Tsai HF, Wang YA (2016) Optimized multiple platforms for big data analysis. In: 2016 IEEE second international conference on multimedia Big Data (BigMM). IEEE, pp 155–158. https://doi.org/10.1109/BigMM.2016.61

  25. Chen Q, Zhang D, Guo M, Deng Q, Guo S (2010) Samr: a self-adaptive mapreduce scheduling algorithm in heterogeneous environment. In: 2010 10th IEEE international conference on computer and information technology. IEEE, pp 2736–2743. https://doi.org/10.1109/CIT.2010.458

  26. Cooper BF, Silberstein A, Tam E, Ramakrishnan R, Sears R (2010) Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st ACM symposium on cloud computing. ACM, pp 143–154. https://doi.org/10.1145/1807128.1807152

  27. Dede E, Fadika Z, Govindaraju M, Ramakrishnan L (2014) Benchmarking MapReduce implementations under different application scenarios. Future Gener Comput Syst 36:389–399. https://doi.org/10.1016/j.future.2014.01.001

    Article  Google Scholar 

  28. Dharanipragada J, Padala S, Kammili B, Kumar V (2017) Tula: a disk latency aware balancing and block placement strategy for Hadoop. In: 2017 IEEE international conference on Big Data (Big Data). IEEE, pp 2853–2858. https://doi.org/10.1109/BigData.2017.8258253

  29. Echihabi K, Zoumpatianos K, Palpanas T (2020) Big sequence management: on scalability. In: Proceedings of the IEEE international conference on Big Data. IEEE BigData

  30. Elmubarak SA, Yousif A, Bashir MB (2017) Performance based ranking model for cloud SaaS services. Int J Inf Technol Comput Sci 9(1):65–71. https://doi.org/10.5815/ijitcs.2017.01.08

    Article  Google Scholar 

  31. Ferdman M, Adileh A, Kocberber O, Volos S, Alisafaee M, Jevdjic D, Kaynak C, Popescu AD, Ailamaki A, Falsafi B (2012) Clearing the clouds: a study of emerging scale-out workloads on modern hardware. In: ACM SIGPLAN Notices, vol 47. ACM, pp 37–48. https://doi.org/10.1145/2150976.2150982

  32. Gao J, Manjula K, Roopa P, Sumalatha E, Bai X, Tsai WT, Uehara T (2012) A cloud-based TaaS infrastructure with tools for SaaS validation, performance and scalability evaluation. In: 4th IEEE international conference on cloud computing technology and science proceedings. IEEE, pp 464–471. https://doi.org/10.1109/CloudCom.2012.6427555

  33. Gao J, Pattabhiraman P, Bai X, Tsai WT (2011) SaaS performance and scalability evaluation in clouds. In: Proceedings of 2011 IEEE 6th international symposium on service oriented system (SOSE). IEEE, pp 61–71. https://doi.org/10.1109/SOSE.2011.6139093

  34. Garate-Escamilla AK, El Hassani AH, Andres E (2019) Big data scalability based on spark machine learning libraries. In: Proceedings of the 2019 3rd international conference on Big Data research, pp 166–171

  35. Garg N, Janakiram D (2018) Sparker: optimizing spark for heterogeneous clusters. In: 2018 IEEE international conference on cloud computing technology and science (CloudCom). IEEE, pp 1–8. https://doi.org/10.1109/CloudCom2018.2018.00017

  36. Ghasemi E, Chow P (2016) Accelerating Apache Spark big data analysis with fpgas. In: 2016 Intl IEEE conferences on ubiquitous intelligence & computing, advanced and trusted computing, scalable computing and communications, cloud and Big Data computing, internet of people, and smart world congress (UIC/ATC/ScalCom/CBDCom/IoP/SmartWorld). IEEE, pp 737–744. https://doi.org/10.1109/UIC-ATC-ScalCom-CBDCom-IoP-SmartWorld.2016.0119

  37. Ghazal A, Rabl T, Hu M, Raab F, Poess M, Crolotte A, Jacobsen HA (2013) BigBench: Towards an industry standard benchmark for big data analytics. In: Proceedings of the 2013 ACM SIGMOD international conference on Management of data. ACM, pp 1197–1208. https://doi.org/10.1145/2463676.2463712

  38. Govindaraju V, Idicula S, Agrawal S, Vardarajan V, Raghavan A, Wen J, Balkesen C, Giannikis G, Agarwal N, Sedlar E (2017) Big data processing: scalability with extreme single-node performance. In: 2017 IEEE international congress on Big Data (BigData Congress). IEEE, pp 129–136. https://doi.org/10.1109/BigDataCongress.2017.26

  39. Grama A, Gupta A, Kumar V (1996) Isoefficiency function: a scalability metric for parallel algorithms and architectures. IEEE Trans Parallel Distrib Syst 4(8):12–21

    Google Scholar 

  40. Grama AY, Gupta A, Kumar V (1993) Isoefficiency: measuring the scalability of parallel algorithms and architectures. IEEE Parallel Distrib Technol Syst Appl 1(3):12–21. https://doi.org/10.1109/88.242438

    Article  Google Scholar 

  41. Gunther N, Puglia P, Tomasette K (2015) Hadoop superlinear scalability. Queue 13(5):20. https://doi.org/10.1145/2773212.2789974

    Article  Google Scholar 

  42. Guo Y, Rao J, Cheng D, Zhou X (2016) Ishuffle: improving hadoop performance with shuffle-on-write. IEEE Trans Parallel Distrib Syst 28(6):1649–1662. https://doi.org/10.1109/TPDS.2016.2587645

    Article  Google Scholar 

  43. Gustafson JL (1988) Reevaluating Amdahl’s law. Commun ACM 31(5):532–533. https://doi.org/10.1145/42411.42415

    Article  Google Scholar 

  44. Henning S, Hasselbring W (2021) How to measure scalability of distributed stream processing engines? In: Companion of the ACM/SPEC international conference on performance engineering, pp 85–88

  45. Huang S, Huang J, Dai J, Xie T, Huang B (2010) The HiBench benchmark suite: characterization of the MapReduce-based data analysis. In: 2010 IEEE 26th international conference on data engineering workshops (ICDEW 2010). IEEE, pp 41–51 (2010). https://doi.org/10.1109/ICDEW.2010.5452747

  46. Hwang K, Bai X, Shi Y, Li M, Chen WG, Wu Y (2015) Cloud performance modeling with benchmark evaluation of elastic scaling strategies. IEEE Trans Parallel Distrib Syst 27(1):130–143. https://doi.org/10.1109/TPDS.2015.2398438

    Article  Google Scholar 

  47. Hwang K, Shi Y, Bai X (2014) Scale-out vs. scale-up techniques for cloud performance and productivity. In: 2014 IEEE 6th international conference on cloud computing technology and science. IEEE, pp 763–768. https://doi.org/10.1109/CloudCom.2014.66

  48. Iosup A, Epema D (2006) Grenchmark: a framework for analyzing, testing, and comparing grids. In: Sixth IEEE international symposium on cluster computing and the grid (CCGRID’06), vol 1. IEEE, pp 313–320. https://doi.org/10.1109/CCGRID.2006.49

  49. Jiang C, Fan T, Gao H, Shi W, Liu L, Cerin C, Wan J (2020) Energy aware edge computing: a survey. Comput Commun 151:556–580

    Article  Google Scholar 

  50. Jiang C, Fan T, Qiu Y, Wu H, Zhang J, Xiong N, Wan J (2018) Interdomain I/O optimization in virtualized sensor networks. Sensors 18(12):4395. https://doi.org/10.3390/s18124395

    Article  Google Scholar 

  51. Jiang C, Han G, Lin J, Jia G, Shi W, Wan J (2019) Characteristics of co-allocated online services and batch jobs in internet data centers: a case study from Alibaba cloud. IEEE Access 7:22495–22508

    Article  Google Scholar 

  52. Jiang C, Qiu Y, Shi W, Ge Z, Wang J, Chen S, Cerin C, Ren Z, Xu G, Lin J (2020) Characterizing co-located workloads in Alibaba cloud datacenters. IEEE Trans Cloud Comput

  53. Jiang C, Wang Y, Ou D, Li Y, Zhang J, Wan J, Luo B, Shi W (2017) Energy efficiency comparison of hypervisors. Sustain Comput Inform Syst. https://doi.org/10.1016/j.suscom.2017.09.005

  54. Jiang C, Wang Y, Ou D, Luo B, Shi W (2017) Energy proportional servers: where are we in 2016? In: 2017 IEEE 37th international conference on distributed computing systems (ICDCS). IEEE, pp 1649–1660. https://doi.org/10.1109/ICDCS.2017.285

  55. Jiang C, Wang Y, Ou D, Qiu Y, Li Y, Wan J, Luo B, Shi W, Cerin C (2018) Ease: energy efficiency and proportionality aware virtual machine scheduling. In: 2018 30th international symposium on computer architecture and high performance computing (SBAC-PAD). IEEE, pp 65–68

  56. Jogalekar P, Woodside M (2000) Evaluating the scalability of distributed systems. IEEE Trans Parallel Distrib Syst 11(6):589–603. https://doi.org/10.1109/71.862209

    Article  Google Scholar 

  57. Kim K, Jeon K, Han H, Kim S.g, Jung H, Yeom HY (2008) Mrbench: a benchmark for MapReduce framework. In: 2008 14th IEEE international conference on parallel and distributed systems. IEEE, pp 11–18. https://doi.org/10.1109/ICPADS.2008.70

  58. Lee JY, Lee JW, Kim SD, et al (2009) A quality model for evaluating software-as-a-service in cloud computing. In: 2009 seventh ACIS international conference on software engineering research, management and applications. IEEE, pp 261–266. https://doi.org/10.1109/SERA.2009.43

  59. Li M, Tan J, Wang Y, Zhang L, Salapura V (2015) Sparkbench: a comprehensive benchmarking suite for in memory data analytic platform Spark. In: Proceedings of the 12th ACM international conference on computing frontiers. ACM, p 53. https://doi.org/10.1145/2742854.2747283

  60. Li Z, Shen H (2017) Measuring scale-up and scale-out Hadoop with remote and local file systems and selecting the best platform. IEEE Trans Parallel Distrib Syst 28(11):3201–3214. https://doi.org/10.1109/TPDS.2017.2712635

    Article  Google Scholar 

  61. Lin J (2018) Scale up or scale out for graph processing? IEEE Internet Comput 22(3):72–78. https://doi.org/10.1109/MIC.2018.032501520

    Article  Google Scholar 

  62. Marco VS, Taylor B, Porter B, Wang Z (2017) Improving Spark application throughput via memory aware task co-location: a mixture of experts approach. In: Proceedings of the 18th ACM/IFIP/USENIX middleware conference. ACM, pp 95–108. https://doi.org/10.1145/3135974.3135984

  63. Meena M, Bharadi VA (2016) Performance analysis of cloud based software as a service (SaaS) model on public and hybrid cloud. In: 2016 symposium on colossal data analysis and networking (CDAN). IEEE, pp 1–6. https://doi.org/10.1109/CDAN.2016.7570951

  64. Meng H, Yu S, Liu F, Xiao N (2017) Research on memory management and cache replacement policies in Spark. Comput Sci 44(6):31–35. https://doi.org/10.11896/j.issn.1002-137X.2017.06.005

    Article  Google Scholar 

  65. Ming Z, Luo, C, Gao W, Han R, Yang Q, Wang L, Zhan J (2013) BDGS: a scalable big data generator suite in big data benchmarking. In: Advancing Big Data benchmarks. Springer, pp 138–154. https://doi.org/10.1007/978-3-319-10596-3_11

  66. Nguyen N, Khan MMH, Albayram Y, Wang K (2017) Understanding the influence of configuration settings: an execution model-driven framework for Apache Spark platform. In: 2017 IEEE 10th international conference on cloud computing (CLOUD). IEEE, pp 802–807. https://doi.org/10.1109/CLOUD.2017.119

  67. Nguyen N, Khan MMH, Wang K (2016) Csminer: an automated tool for analyzing changes in configuration settings across multiple versions of large scale cloud software. In: 2016 IEEE 9th international conference on cloud computing (CLOUD). IEEE, pp 472–480. https://doi.org/10.1109/CLOUD.2016.0069

  68. Ousterhout K, Rasti R, Ratnasamy S, Shenker S, Chun BG (2015) Making sense of performance in data analytics frameworks. In: 12th \(\{\)USENIX\(\}\) symposium on networked systems design and implementation (\(\{\)NSDI\(\}\) 15), pp 293–307

  69. Pirzadeh P, Carey M, Westmann T (2017) A performance study of big data analytics platforms. In: 2017 IEEE international conference on Big Data (Big Data). IEEE, pp 2911–2920. https://doi.org/10.1109/BigData.2017.8258260

  70. Qiu Y, Jiang C, Wang Y, Ou D, Li Y, Wan J (2019) Energy aware virtual machine scheduling in data centers. Energies 12(4):646. https://doi.org/10.3390/en12040646

    Article  Google Scholar 

  71. Raïs I, Balouek-Thomert D, Orgerie A.C, Lefèvre L, Parashar M (2019) Leveraging energy-efficient non-lossy compression for data-intensive applications. In: 2019 international conference on high performance computing & simulation (HPCS). IEEE

  72. Ruan X, Chen H (2017) Improving Shuffle I/O performance for big data processing using hybrid storage. In: 2017 international conference on computing, networking and communications (ICNC). IEEE, pp 476–480. https://doi.org/10.1109/ICCNC.2017.7876175

  73. Sandel R, Shtern M, Fokaefs M, Litoiu M (2015) Evaluating cluster configurations for big data processing: an exploratory study. In: 2015 IEEE 9th international symposium on the maintenance and evolution of service-oriented and cloud-based environments (MESOCA). IEEE, pp 23–30. https://doi.org/10.1109/MESOCA.2015.7328122

  74. Siegmund N, Grebhahn A, Apel S, Kästner C (2015) Performance-influence models for highly configurable systems. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering. ACM, pp 284–294. https://doi.org/10.1145/2786805.2786845

  75. Sun X-H, Rover DT (1994) Scalability of parallel algorithm-machine combinations. IEEE Trans Parallel Distrib Syst 5(6):599–613. https://doi.org/10.1109/71.285606

    Article  Google Scholar 

  76. Tsai WT, Huang Y, Shao Q (2011) Testing the scalability of SaaS applications. In: 2011 IEEE international conference on service-oriented computing and applications (SOCA). IEEE, pp 1–4. https://doi.org/10.1109/SOCA.2011.6166245

  77. Wang G, Xu J, He B (2016) A novel method for tuning configuration parameters of Spark based on machine learning. In: 2016 IEEE 18th international conference on high performance computing and communications; IEEE 14th international conference on smart city; IEEE 2nd international conference on data science and systems (HPCC/SmartCity/DSS). IEEE, pp 586–593. https://doi.org/10.1109/HPCC-SmartCity-DSS.2016.0088

  78. Wang K, Khan MMH (2015) Performance prediction for Apache Spark platform. In: 2015 IEEE 17th international conference on high performance computing and communications, 2015 IEEE 7th international symposium on cyberspace safety and security, and 2015 IEEE 12th international conference on embedded software and systems. IEEE, pp 166–173. https://doi.org/10.1109/HPCC-CSS-ICESS.2015.246

  79. Wang K, Khan MMH, Nguyen N, Gokhale S (2016) Modeling interference for Apache Spark jobs. In: 2016 IEEE 9th international conference on cloud computing (CLOUD). IEEE, pp 423–431. https://doi.org/10.1109/CLOUD.2016.0063

  80. Wang L, Zhan J, Gao W, Jiang Z, Ren R, He X, Luo C, Lu G, Li J (2018) BOPS, not FLOPS! A new metric and roofline performance model for datacenter computing. arXiv preprint arXiv:1801.09212

  81. Wang L, Zhan J, Luo C, Zhu Y, Yang Q, He Y, Gao W, Jia Z, Shi Y, Zhang S, et al (2014) Bigdatabench: a big data benchmark suite from internet services. In: 2014 IEEE 20th international symposium on high performance computer architecture (HPCA). IEEE, pp 488–499. https://doi.org/10.1109/HPCA.2014.6835958

  82. Xia Y, Yang F (2017/04) Locality-based partitioning for Spark. In: 2017 5th international conference on frontiers of manufacturing science and measuring technology (FMSMT 2017). Atlantis Press. https://doi.org/10.2991/fmsmt-17.2017.233

  83. Xie R, Jia X (2015) Data transfer scheduling for maximizing throughput of big-data computing in cloud systems. IEEE Trans Cloud Comput 6(1):87–98. https://doi.org/10.1109/TCC.2015.2464808

    Article  Google Scholar 

  84. Xu L, Li M, Zhang L, Butt AR, Wang Y, Hu ZZ (2016) MEMTUNE: dynamic memory management for in-memory data analytic platforms. In: 2016 IEEE international parallel and distributed processing symposium (IPDPS), pp 383–392. IEEE. https://doi.org/10.1109/IPDPS.2016.105

  85. Yigitbasi N, Iosup A, Epema D, Ostermann S (2009) C-meter: a framework for performance analysis of computing clouds. In: 2009 9th IEEE/ACM international symposium on cluster computing and the grid. IEEE, pp 472–477. https://doi.org/10.1109/CCGRID.2009.40

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (No. 61972118), the Science and Technology Project of State Grid Corporation of China (Research and Application on Multi-Datacenters Cooperation & Intelligent Operation and Maintenance, No. 5700-202018194A-0-0-00),and the the Science and Technology Project of State Grid Corporation of China (No. SGSDXT00DKJS1900040).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, 310018, China

    Yuegang Li, Dongyang Ou, Xin Zhou & Congfeng Jiang

  2. Université Sorbonne Paris Nord, LIPN UMR CNRS 7030, 93430, Villetaneuse, France

    Christophe Cérin

Authors
  1. Yuegang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dongyang Ou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Congfeng Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christophe Cérin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Congfeng Jiang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Ou, D., Zhou, X. et al. Scalability and performance analysis of BDPS in clouds. Computing 104, 1425–1460 (2022). https://doi.org/10.1007/s00607-022-01056-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-022-01056-7

Keywords

Mathematics Subject Classification

Search

Navigation