iCiRe: Optimal Scheduling of HPC Applications in Multi-Cloud
Article No.: 48, Pages 1 - 6
Abstract
High Performance Computing (HPC) applications are resource-intensive requiring high-end machines for computation and data transfer operations. The data center resources are over-provisioned to address peak workload and high-performance requirements. However, idle resources during non-peak hours result in cost escalations. Many organizations are using cloud-bursting solutions to address these challenges. Moreover, mapping these applications to optimal cloud instances resulting in cost-effective and high-performance deployment is a challenging task due to a large number of resource types from multiple cloud vendors.
In this work, we present an intelligent Cloud instance Recommender engine (iCiRe) framework for finding the optimal cloud instances for HPC workloads in a multi-cloud environment. The framework consists of a cloud suitability finder, a performance analyzer, and a decision-making enabler module. Our approach ensures that the HPC application workload is assessed for its suitability for the cloud prior to migration. Additionally, we evaluate the proposed framework using two HPC applications from the finance and life-sciences domain.
References
[1]
Amazon. 2023. AWS EC2. Accessed September 21, 2023. https://aws.amazon.com/ec2/
[2]
Amazon. 2023. AWS HPC. Accessed September 21, 2023. https://aws.amazon.com/hpc/
[3]
Azure. 2023. Azure VM. Accessed September 21, 2023. https://azure.microsoft.com/en-in/products/virtual-machines
[4]
Jeferson R. Brunetta and Edson Borin. 2019. Selecting Efficient Cloud Resources for HPC Workloads. In Proceedings of the 12th IEEE/ACM International Conference on Utility and Cloud Computing (Auckland, New Zealand) (UCC'19). Association for Computing Machinery, New York, NY, USA, 155--164.
[5]
Dheeraj Chahal, Pradeep Gameria, Rajesh Kulkarni, and Amit Kalele. 2022. iSeSA: Towards Migrating HPC and AI Workloads to Serverless Platform. In Proceedings of the 12th Workshop on AI and Scientific Computing at Scale using Flexible Computing Infrastructures. 1--8.
[6]
Collectl. 2023. Linux Collectl. http://collectl.sourceforge.net
[7]
Darshan. 2023. Darshan: HPC I/O characterization tool. https://www.mcs.anl.gov/research/projects/darshan/
[8]
Ivan Dokmanic, Reza Parhizkar, Juri Ranieri, and Martin Vetterli. 2015. Euclidean distance matrices: essential theory, algorithms, and applications. IEEE Signal Processing Magazine 32, 6 (2015), 12--30.
[9]
Nicolas Dube, Duncan Roweth, Paolo Faraboschi, and Dejan Milojicic. 2021. Future of HPC: The Internet of Workflows. IEEE Internet Computing 25, 5 (2021), 26--34.
[10]
Centor for Computational Structural Biology. 2023. AutoDock. https://autodock.scripps.edu/
[11]
Google. 2023. Compute Engine. Accessed September 21, 2023. https://cloud.google.com/compute
[12]
Stefan Kehrer and Wolfgang Blochinger. 2019. Migrating parallel applications to the cloud: assessing cloud readiness based on parallel design decisions. SICS Software-Intensive Cyber-Physical Systems 34 (06 2019).
[13]
Stefan Kehrer and Wolfgang Blochinger. 2019. A survey on cloud migration strategies for high performance computing. In Proceedings of the 13th Symposium and Summer School on Service-Oriented Computing (SummerSoc19). IBM Research Division, 57--69.
[14]
Alfirna Rizqi Lahitani, Adhistya Erna Permanasari, and Noor Akhmad Setiawan. 2016. Cosine similarity to determine similarity measure: Study case in online essay assessment. In 2016 4th International Conference on Cyber and IT Service Management. IEEE, 1--6.
[15]
Mariano Ezequiel Mirabelli. 2020. Exploring serverless computing throughout the replica exchange algorithm. (2020).
[16]
V. Munhoz, M. Castro, and O. Mendizabal. 2022. Strategies for Fault-Tolerant Tightly-Coupled HPC Workloads Running on Low-Budget Spot Cloud Infrastructures. In 2022 IEEE 34th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). IEEE Computer Society, Los Alamitos, CA, USA, 263--272.
[17]
Davit Petrosyan and Hrachya Astsatryan. 2022. Serverless High-Performance Computing over Cloud. Cybernetics and Information Technologies 22, 3 (2022), 82--92.
[18]
Penguin Solutions. 2023. Accessed September 21, 2023. https://www.penguinsolutions.com/
[19]
R SYSTEMS. 2023. Accessed September 21, 2023. https://rsystemsinc.com/
[20]
Sam Weekly, Zoey Mertes, and Alex Younts. 2021. Rapid Prototype for Shifting HPC to the Cloud. In 2021 IEEE 17th International Conference on eScience (eScience). IEEE, 262--263.
[21]
Perf Wiki. 2023. Linux Perf. https://perf.wiki.kernel.org/index.php/Main_Page
Index Terms
- iCiRe: Optimal Scheduling of HPC Applications in Multi-Cloud
Index terms have been assigned to the content through auto-classification.
Recommendations
HPC in Weather Forecast: Moving to the Cloud
Many traditional High-Performance Computing systems rely on hardware performance as well as compiler parallelization. The increased demand on computing resources rapidly escalates costs, which are the bigger limitation nowadays in both public and ...
Optimal Resource Usage in Multi-Cloud Computing Environment
Cloud computing has emerged as a new paradigm for accessing distributed computing resources such as infrastructure, hardware platform, and software applications on-demand over the internet as services. This paper presents an optimal resource management ...
Virtual I/O Forwarding for Cloud-based HPC Applications
ISPA '12: Proceedings of the 2012 IEEE 10th International Symposium on Parallel and Distributed Processing with ApplicationsOver the past years, the cloud, especially Infrastructure as a Service (IaaS) offerings, have seen a staggering rise in popularity due to the convenience and the economic pricing models. Even cloud service providers are starting to offer custom-sized ...
Comments
Information & Contributors
Information
Published In
December 2023
502 pages
ISBN:9798400702341
DOI:10.1145/3603166
- Co-chairs:
- Omer Rana,
- Massimo Villari,
- Program Co-chairs:
- Song Fu,
- Lorenzo Carnevale
Copyright © 2023 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 April 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
UCC '23
Sponsor:
UCC '23: IEEE/ACM 16th International Conference on Utility and Cloud Computing
December 4 - 7, 2023
Taormina (Messina), Italy
Acceptance Rates
Overall Acceptance Rate 38 of 125 submissions, 30%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 39Total Downloads
- Downloads (Last 12 months)39
- Downloads (Last 6 weeks)2
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in