Abstract
Distributed Computing platforms involve multiple processing systems connected through a network and support the parallel execution of applications. They enable huge computational power and data processing with a quick response time. Examples of use cases requiring distributed computing are stream processing, batch processing, and client-server models. Most of these use cases involve tasks executed in a sequence on different computers to arrive at the results. Numerous distributed computing algorithms have been suggested in the literature, focusing on efficiently utilizing compute nodes to handle tasks within a workflow on on-premises setups. Industries that previously relied on on-premises setups for big data processing are shifting to cloud environments offered by providers such as Azure, Amazon, and Google. This transition is driven by the convenience of Platform-as-a-Service offerings scuh as Batch Services, Hadoop, and Spark. These PaaS services, coupled with auto-provisioning and auto-scaling, reduce costs through a Pay-As-You-Go model. However, a significant challenge with cloud services is configuring them with only a single type of machine for performing all the tasks in the distributed workflow, although each task has diverse compute node requirements. To address this issue in this paper, we propose an Intelligent task scheduling framework that uses a classifier-based dynamic task scheduling approach to determine the best available node for each task. The proposed framework improves the overall performance of the distributed computing workflow by optimizing task allocation and utilization of resources. Although Azure Batch Service is used in this paper to illustrate the proposed framework, our approach can also be implemented on other PaaS distributed computing platforms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Directory of Azure Cloud Services | Microsoft Azure. https://azure.microsoft.com/en-in/products/
Chen, C.-Y., Huang, J.-J.: Double deep autoencoder for heterogeneous distributed clustering. Information 10(4), 144 (2019). https://doi.org/10.3390/info10040144
Pop, D., Iuhasz, G., Petcu, D.: Distributed platforms and cloud services: enabling machine learning for big data. In: Mahmood, Z. (ed.) Data Science and Big Data Computing, pp. 139–159. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-31861-5_7
Nadeem, F., Alghazzawi, D., Mashat, A., Faqeeh, K., Almalaise, A.: Using machine learning ensemble methods to predict execution time of e-science workflows in heterogeneous distributed systems. IEEE Access 7, 25138–25149 (2019). https://doi.org/10.1109/ACCESS.2019.2899985
Sarnovsky, M., Olejnik, M.: Improvement in the efficiency of a distributed multi-label text classification algorithm using infrastructure and task-related data. Informatics 6(12), 1–15 (2019). https://doi.org/10.3390/informatics6010012
Ranjan, R.: Streaming big data processing in datacenter clouds, pp-78–83. IEEE Computer Society (2014)
Al-kahtani, M.S., Karim, L.: An efficient distributed algorithm for big data processing. Arab. J. Sci. Eng. 42(8), 3149–3157 (2017). https://doi.org/10.1007/s13369-016-2405-y
Bahnasawy, N.A., Omara, F., Koutb, M.A., Mosa, M.: Optimization procedure for algorithms of task scheduling in high performance heterogeneous distributed computing systems. Egypt. Inform. J. 12(3), 219–229 (2011). https://doi.org/10.1016/j.eij.2011.10.001. ISSN 1110-8665
Jahanshahi, M., Meybodi, M.R., Dehghan, M.: A new approach for task scheduling in distributed systems using learning automata. In: 2009 IEEE International Conference on Automation and Logistics, pp. 62–67 (2009). https://doi.org/10.1109/ICAL.2009.5262978
Sriraman, A., Dhanotia, A., Wenisch, T.F.: SoftSKU: optimizing server architectures for microservice diversity @scale. In: 2019 ACM/IEEE 46th Annual International Symposium on Computer Architecture (ISCA), pp. 513–526 (2019)
Pandey, R., Silakari, S.: Investigations on optimizing performance of the distributed computing in heterogeneous environment using machine learning technique for large scale data set. Mater. Today: Proc. (2021). https://doi.org/10.1016/j.matpr.2021.07.089. ISSN 2214-7853
Optical character recognition. https://en.wikipedia.org/wiki/Optical_character_recognition
Entity Extraction. https://en.wikipedia.org/wiki/Named-entity_recognition
Directed acyclic graph – Wikipedia. https://en.wikipedia.org/wiki/Directed_acyclic_graph
Scanned Well Files Query. https://www.data.bsee.gov/Other/DiscMediaStore/ScanWellFiles.aspx
Pricing - Windows Virtual Machines | Microsoft Azure. https://azure.microsoft.com/en-in/pricing/details/virtual-machines/windows/
Getting Started with AWS Batch - AWS Batch. https://docs.aws.amazon.com/batch/latest/userguide/Batch_GetStarted.html#first-run-step-2
Batch service on Google Cloud. https://cloud.google.com/blog/products/compute/new-batch-service-processes-batch-jobs-on-google-cloud
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Venkatesh, P.R., Radha Krishna, P. (2024). An Improved and Efficient Distributed Computing Framework with Intelligent Task Scheduling. In: Devismes, S., Mandal, P.S., Saradhi, V.V., Prasad, B., Molla, A.R., Sharma, G. (eds) Distributed Computing and Intelligent Technology. ICDCIT 2024. Lecture Notes in Computer Science, vol 14501. Springer, Cham. https://doi.org/10.1007/978-3-031-50583-6_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-50583-6_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-50582-9
Online ISBN: 978-3-031-50583-6
eBook Packages: Computer ScienceComputer Science (R0)