Abstract
Nowadays, most data analytical applications comprise of multiple tasks, which can be represented as workflow in nature. Some of data analytical applications, the data requests arrived continuously, such as fraud detection application, order application, etc. Generally, such streaming analytical workflow applications have a rigid requirement on throughput. It is critical to provisioning resource for streaming analytical workflows on a cloud platform with financial cost as minimizing as possible while still guaranteeing system throughput. We propose a cost effective resource provisioning algorithm which can guarantee system throughput. Experiments on the Alibaba cloud indicate that our proposed scheduling algorithm can guarantee the workflow throughput under different intensities of the workloads.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Google cloud. https://cloud.google.com/compute/
Amazon: AWS cloud. http://aws.amazon.com/ec2/instance types
Armbrust, M., et al.: A view of cloud computing. Commun. ACM 53(4), 50–58 (2010)
Chen, M., Mao, S., Liu, Y.: Big data: a survey. Mob. Netw. Appl. 19(2), 171–209 (2014)
Chen, W., Paik, I., Hung, P.C.K.: Transformation-based streaming workflow allocation on geo-distributed datacenters for streaming big data processing. IEEE Trans. Serv. Comput. 1 (2017)
Chen, W., Paik, I., Li, Z.: Cost-aware streaming workflow allocation on geo-distributed data centers. IEEE Trans. Comput. 66(2), 256–271 (2017)
Delimitrou, C., Kozyrakis, C.: Paragon: QoS-aware scheduling for heterogeneous datacenters. In: Architectural Support for Programming Languages and Operating Systems, vol. 48, no. 4, pp. 77–88 (2013)
Garey, M.R., Graham, R.L., Johnson, D.S., Yao, A.C.: Resource constrained scheduling as generalized bin packing. J. Comb. Theory Ser. A 21(3), 257–298 (1976)
Guruprasad, H.S., Bhavani, B.H.: Resource provisioning techniques in cloud computing environment: a survey. Int. J. Res. Comput. Commun. Technol. 3, 395–401 (2014)
Hirzel, M., Soulé, R., Schneider, S., Gedik, B., Grimm, R.: A catalog of stream processing optimizations. ACM Comput. Surv. 46(4), 46:1–46:34 (2013)
Khan, S., Shakil, K.A., Alam, M.: Workflow-based big data analytics in the cloud environment present research status and future prospects. CoRR (abs/1711.02087) (2017)
Mars, J., Tang, L., Hundt, R., Skadron, K., Soffa, M.L.: Bubble-up: increasing utilization in modern warehouse scale computers via sensible co-locations. In: 44rd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2011, Porto Alegre, Brazil, 3–7 December 2011, pp. 248–259 (2011)
Raz, T.: The art of computer systems performance analysis: techniques for experimental design, measurement, simulation, and modeling (Raj Jain). SIAM Rev. 34(3), 518–519 (1992)
Rodriguez, M.A., Buyya, R.: A taxonomy and survey on scheduling algorithms for scientific workflows in IaaS cloud computing environments. Concurr. Comput.: Pract. Exp. 29(8), e4041 (2017)
Sandryhaila, A., Moura, J.M.F.: Big data analysis with signal processing on graphs: representation and processing of massive data sets with irregular structure. IEEE Signal Process. Mag. 31(5), 80–90 (2014)
Wen, Y., Chen, Z., Chen, T.: An improved scheduling algorithm for dynamic batch processing in workflows. In: Proceedings of the 2013 International Conference on Cloud and Green Computing, No. 6 in CGC 2013, pp. 502–507 (2013)
Acknowledgments
This research was supported in part by the National Key Research and Development Plan of China (No. 2018YFB1003800), the National Natural Science Foundation of China (No. 61472253, 61772334), and the Cross Research Fund of Biomedical Engineering of Shanghai Jiao Tong University (No. YG2015MS61).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yao, Y., Cao, J., Qian, S. (2019). Throughput-Guarantee Resource Provisioning for Streaming Analytical Workflows in the Cloud. In: Sun, Y., Lu, T., Xie, X., Gao, L., Fan, H. (eds) Computer Supported Cooperative Work and Social Computing. ChineseCSCW 2018. Communications in Computer and Information Science, vol 917. Springer, Singapore. https://doi.org/10.1007/978-981-13-3044-5_16
Download citation
DOI: https://doi.org/10.1007/978-981-13-3044-5_16
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-3043-8
Online ISBN: 978-981-13-3044-5
eBook Packages: Computer ScienceComputer Science (R0)