Abstract
Traditionally, complex engineering applications (CEAs), which consist of numerous components (software) and require a large amount of computing resources, usually run in dedicated clusters or high performance computing (HPC) centers. Nowadays, Cloud computing system with the ability of providing massive computing resources and customizable execution environment is becoming an attractive option for CEAs. As a new type on Cloud applications, CEA also brings the challenges of dealing with Cloud resources. In this paper, we provide a comprehensive survey of Cloud resource management research for CEAs. The survey puts forward two important questions: 1) what are the main challenges for CEAs to run in Clouds? and 2) what are the prior research topics addressing these challenges? We summarize and highlight the main challenges and prior research topics. Our work can be probably helpful to those scientists and engineers who are interested in running CEAs in Cloud environment.
Similar content being viewed by others
References
Crago S, Dunn K, Eads P, Hochstein L, Kang D I, Kang M, Modium D, Singh K, Suh J, Walters J P. Heterogeneous cloud computing. In: Proceedings of 2011 IEEE International Conference on Cluster Computing (CLUSTER). 2011, 378–385
Arian E. On the coupling of aerodynamic and structural design. Journal of Computational Physics, 1997, 135(1): 83–96
Wang C, Xu L. Parameter mapping and data transformation for engineering application integration. Information Systems Frontiers, 2008, 10(5): 589–600
Ong M, Thompson H. Challenges for wireless sensing in complex engineering applications. In: Proceedings of the 37th Annual Conference on IEEE Industrial Electronics Society (IECON). 2011, 2106–2111
Bichon B, Eldred M, Swiler L, Mahadevan S, McFarland J. Multimodal reliability assessment for complex engineering applications using efficient global optimization. In: Proceedings of the 48th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, AIAA-2007–1946. 2007, 3029–3040
Chacón Rebollo T, Gómez Mármol M, Restelli M. Numerical analysis of penalty stabilized finite element discretizations of evolution navier–stokes equations. Journal of Scientific Computing, 2015, 63(3): 885–912
Campana E F, Liuzzi G, Lucidi S, Peri D, Piccialli V, Pinto A. New global optimization methods for ship design problems. Optimization and Engineering, 2009, 10(4): 533–555
Jin C, Wang Y, Zhang W, Lin Y. Study on semi-finished ship structural components assembly sequence optimization. In: Proceedings of the 6th International Conference on Natural Computation (ICNC). 2010, 2706–2709
Kwon S, Kim B C, Mun D, Han S. Simplification of feature-based 3D CAD assembly data of ship and offshore equipment using quantitative evaluation metrics. Computer-Aided Design, 2015, 59: 140–154
Palankar M R, Iamnitchi A, Ripeanu M, Garfinkel S. Amazon S3 for science grids: a viable solution? In: Proceedings of the 2008 International Workshop on Data-aware Distributed Computing. 2008, 55–64
Hazelhurst S. Scientific computing using virtual high-performance computing: a case study using the amazon elastic computing cloud. In: Proceedings of the 2008 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on IT Research in Developing Countries: Riding the Wave of Technology. 2008, 94–103
Vöckler J S, Juve G, Deelman E, Rynge M, Berriman B. Experiences using cloud computing for a scientific workflow application. In: Proceedings of the 2nd International Workshop on Scientific Cloud Computing. 2011, 15–24
Juve G, Deelman E, Vahi K, Mehta G, Berriman B, Berman B P, Maechling P. Scientific workflow applications on amazon EC2. In: Proceedings of the 5th IEEE International Conference on E-ScienceWorkshops. 2009, 59–66
Rehr J J, Vila F D, Gardner J P, Svec L, Prange M. Scientific computing in the cloud. Computing in Science & Engineering, 2010, 12(3): 34–43
Lin G, Han B, Yin J, Gorton I. Exploring cloud computing for largescale scientific applications. In: Proceedings of the 9th IEEE World Congress on Services (SERVICES). 2013, 37–43
Ramakrishnan L, Zbiegel P T, Campbell S, Bradshaw R, Canon R S, Coghlan S, Sakrejda I, Desai N, Declerck T, Liu A. Magellan: experiences from a science cloud. In: Proceedings of the 2nd International Workshop on Scientific Cloud Computing. 2011, 49–58
Georgescu S, Chow P. GPU accelerated CAE using open solvers and the cloud. ACMSIGARCH Computer Architecture News, 2011, 39(4): 14–19
Zhai Y, Liu M, Zhai J, Ma X, Chen W. Cloud versus in-house cluster: evaluating amazon cluster compute instances for running MPI applications. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC). 2011, 11
Janapa Reddi V, Lee B C, Chilimbi T, Vaid K. Web search using mobile cores: quantifying and mitigating the price of efficiency. ACM SIGARCH Computer Architecture News, 2010, 38(3): 314–325
Lim K, Ranganathan P, Chang J, Patel C, Mudge T, Reinhardt S. Understanding and designing new server architectures for emerging warehouse-computing environments. In: Proceedings of the 35th International Symposium on Computer Architecture (ISCA). 2008, 315–326
Santos J R, Turner Y, Janakiraman G J, Pratt I. Bridging the gap between software and hardware techniques for I/O virtualization. In: Proceedings of the USENIX Annual Technical Conference (ATC). 2008, 29–42
Ram K K, Santos J R, Turner Y, Cox A L, Rixner S. Achieving 10 Gb/s using safe and transparent network interface virtualization. In: Proceedings of the ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE). 2009, 61–70
Chen H, Wu S, Shi X, Jin H, Fu Y. LCM: a lightweight communication mechanism in HPC cloud. In: Proceedings of the 6th International Conference on Pervasive Computing and Applications. 2011, 443–451
Ram K K, Santos J R, Turner Y. Redesigning Xen’s memory sharing mechanism for safe and efficient I/O virtualization. In: Proceedings of the 2nd conference on I/O virtualization. 2010
Jian Z, Xiaoyong L, Haibing G. The optimization of Xen network virtualization. In: Proceedings of the International Conference on Computer Science and Software Engineering. 2008, 431–436
Guo D, Liao G, Bhuyan L N. Performance characterization and cacheaware core scheduling in a virtualized multi-core server under 10GbE. In: Proceedings of the IEEE International Symposium on Workload Characterization. 2009, 168–177
Liao G, Guo D, Bhuyan L, King S R. Software techniques to improve virtualized I/O performance on multi-core systems. In: Proceedings of the 4th ACM/IEEE Symposium on Architectures for Networking and Communications Systems. 2008, 161–170
Gordon A, Amit N, Har’El N, Ben-Yehuda M, Landau A, Schuster A, Tsafrir D. ELI: bare-metal performance for I/O virtualization. In: Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 2012, 411–422
Abramson D. Intel virtualization technology for directed I/O. Intel Technology Journal, 2006, 10(3): 179–192
Rixner S. Network virtualization: breaking the performance barrier. Queue, 2008, 6(1): 37
Willmann P, Shafer J, Carr D, Menon A, Rixner S, Cox A L, Zwaenepoel W. Concurrent direct network access for virtual machine monitors. In: Proccedings of the 13th IEEE International Symposium on High Performance Computer Architecture (HPCA). 2007, 306–317
Liu J. Evaluating standard-based self-virtualizing devices: a performance study on 10 GbE NICs with SR-IOV support. In: Proceedings of the IEEE International Symposium on Parallel & Distributed Processing (IPDPS). 2010, 1–12
Jones S T, Arpaci-Dusseau A C, Arpaci-Dusseau R H. Antfarm: tracking processes in a virtual machine environment. In: Proceedings of the USENIX Annual Technical Conference (ATC). 2006, 1–14
Jin H, Ling X, Ibrahim S, Cao W, Wu S, Antoniu G. Flubber: two-level disk scheduling in virtualized environment. Future Generation Computer Systems, 2013, 29(8): 2222–2238
Xu Y, Jiang S. A scheduling framework that makes any disk schedulers non-work-conserving solely based on request characteristics. In: Proceedings of the USENIX Conference on File and Storage Technologies. 2011, 119–132
Zhang B B, Wang X L, Yang L, Lai R F, Wang Z L, Luo Y W, Li X M. Modifying guest OS to optimize I/O virtualization in KVM. Chinese Journal of Computers, 2010, 33(12): 2312–2319
Ling X, Ibrahim S, Jin H, Wu S, Tao S. Exploiting spatial locality to improve disk efficiency in virtualized environments. In: Proceedings of the 21st IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS). 2013
Wang H, Varman P. A flexible approach to efficient resource sharing in virtualized environments. In: Proceedings of the 8th ACM International Conference on Computing Frontiers. 2011, 1–10
Seelam S R, Teller P J. Virtual I/O scheduler: a scheduler of schedulers for performance virtualization. In: Proceedings of the 3rd International Conference on Virtual Execution Environments. 2007, 105–115
Kesavan M, Gavrilovska A, Schwan K. Differential virtual time (DVT): rethinking I/O service differentiation for virtual machines. In: Proceedings of the 1st ACM Symposium on Cloud Computing. 2010, 27–38
Gulati A, Ahmad I, Waldspurger C A. PARDA: proportional allocation of resources for distributed storage access. In: Proccedings of the 7th Conference on File and Storage Technologies (FAST). 2009, 85–98
Gulati A, Merchant A, Varman P J. mClock: handling throughput variability for hypervisor I/O scheduling. In: Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). 2010, 1–7
Arunagiri S, Kwok Y, Teller P J, Portillo R A, Seelam S R. FAIRIO: a throughput-oriented algorithm for differentiated I/O performance. International Journal of Parallel Programming, 2014, 42(1): 165–197
Shue D, Freedman M J, Shaikh A. Performance isolation and fairness for multi-tenant cloud storage. In: Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI). 2012, 349–362
Lin C, Lu S. Scheduling scientific workflows elastically for cloud computing. In: Proceedings of the IEEE International Conference on Cloud Computing (CLOUD). 2011, 746–747
Zhang F, Cao J, Hwang K, Wu C. Ordinal optimized scheduling of scientific workflows in elastic compute clouds. In: Proceedings of the 3rd IEEE International Conference on Cloud Computing Technology and Science (CloudCom). 2011, 9–17
Rahman M, Hassan R, Ranjan R, Buyya R. Adaptive workflow scheduling for dynamic grid and cloud computing environment. Concurrency and Computation: Practice and Experience, 2013, 25(13): 1816–1842
Liu B, Li J, Liu C. Cloud-based bioinformatics workflow platform for large-scale next-generation sequencing analyses. Journal of Biomedical Informatics, 2014, 49: 119–133
Liu S W, Kong L M, Ren K J, Song J Q, Deng K F, Leng H Z. A twostep data placement and task scheduling strategy for optimizing scientific workflow performance on cloud computing platform. Chinese Journal of Computers, 2011, 34(11): 2121–2130
Deelman E, Chervenak A. Data management challenges of dataintensive scientific workflows. In: Proceedings of the 8th IEEE International Symposium on Cluster Computing and the Grid (CCGRID). 2008, 687–692
Yuan D, Yang Y, Liu X, Chen J. A cost-effective strategy for intermediate data storage in scientific cloud workflow systems. In: Proceedings of 2010 IEEE International Symposium on Parallel and Distributed Processing (IPDPS). 2010, 1–12
Zheng P, Cui L Z, Wang H Y, Xu M. A data placement strategy for dataintensive applications in cloud. Chinese Journal of Computers, 2010, 33(8): 1472–1480
He B, Fang W, Luo Q, Govindaraju N K, Wang T. Mars: a Map Reduce framework on graphics processors. In: Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques (PACT). 2008, 260–269
Linderman M D, Collins J D, Wang H, Meng T H. Merge: a programming model for heterogeneous multi-core systems. In: Proceedings of the 13th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). 2008, 287–296
Boob S, Gonzalez-Velez H, Popescu A M. Automated instantiation of heterogeneous fast flow CPU/GPU parallel pattern applications in clouds. In: Proceedings of the 22nd Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP). 2014, 162–169
Campa S, Danelutto M, Goli M, González-Vélez H, Popescu A M, Torquati M. Parallel patterns for heterogeneous CPU/GPU architectures: structured parallelism from cluster to cloud. Future Generation Computer Systems, 2014, 37: 354–366
Luk C K, Hong S, Kim H. Qilin: exploiting parallelism on heterogeneous multiprocessors with adaptive mapping. In: Proceedings of the 42nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 2009, 45–55
Ravi V T, Ma W, Chiu D, Agrawal G. Compiler and runtime support for enabling generalized reduction computations on heterogeneous parallel configurations. In: Proceedings of the 24th ACM International Conference on Supercomputing (ICS). 2010, 137–146
Grewe D, O’Boyle M F. A static task partitioning approach for heterogeneous systems using Open CL. Lecture Notes in Computer Science. 2011, 6601: 286–305
Gupta V, Gavrilovska A, Schwan K, Kharche H, Tolia N, Talwar V, Ranganathan P. GViM: GPU-accelerated virtual machines. In: Proceedings of the 3rd ACM Workshop on System-level Virtualization for High Performance Computing. 2009, 17–24
Shi L, Chen H, Sun J, Li K. vCUDA: GPU-accelerated highperformance computing in virtual machines. IEEE Transactions on Computers, 2012, 61(6): 804–816
Giunta G, Montella R, Agrillo G, Coviello G. A GPGPU transparent virtualization component for high performance computing clouds. In: Proceedings of Euro-Par 2010-Parallel Processing. 2010, 379–391
Jo H, Jeong J, Lee M, Choi D H. Exploiting GPUs in virtual machine for Bio Cloud. BioMed Research International, 2013
Shih C S, Wei JW, Hung S H, Chen J, Chang N. Fairness scheduler for virtual machines on heterogonous multi-core platforms. ACM SIGAPP Applied Computing Review, 2013, 13(1): 28–40
Hu L, Che X, Xie Z. GPGPU cloud: a paradigm for general purpose computing. Tsinghua Science and Technology, 2013, 18(1): 22–23
Chen H, Shi L, Sun J. VMRPC: a high efficiency and light weight RPC system for virtual machines. In: Proceedings of the 18th IEEE International Workshop on Quality of Service (IWQoS). 2010
Montella R, Giunta G, Laccetti G. Virtualizing high-end GPGPUs on ARM clusters for the next generation of high performance cloud computing. Cluster Computing, 2014, 17(1): 139–152
Cai Y, Li G, Wang H, Zheng G, Lin S. Development of parallel explicit finite element sheet forming simulation system based on GPU architecture. Advances in Engineering Software, 2012, 45(1): 370–379
Ari I, Muhtaroglu N. Design and implementation of a cloud computing service for finite element analysis. Advances in Engineering Software, 2013, 60: 122–135
Negrut D, Tasora A, Anitescu M, Mazhar H, Heyn T, Pazouki A. Solving large multi-body dynamics problems on the GPU. GPU Gems, 2011, 4: 269–280
Hanniel I, Haller K. Direct rendering of solid CAD models on the GPU. In: Proceedings of the 12th International Conference on Computer-Aided Design and Computer Graphics (CAD/Graphics). 2011, 25–32
Hsieh H T, Chu C H. Particle swarm optimisation (PSO)-based tool path planning for 5-axis flank milling accelerated by graphics processing unit (GPU). International Journal of Computer Integrated Manufacturing, 2011, 24(7): 676–687
Hung Y, Wang W. Accelerating parallel particle swarm optimization via GPU. Optimization Methods and Software, 2012, 27(1): 33–51
Jung H Y, Jun C W, Sohn J H. GPU-based collision analysis between a multi-body system and numerous particles. Journal of Mechanical Science and Technology, 2013, 27(4): 973–980
Nguyen Van H, Dang Tran F, Menaud JM. Autonomic virtual resource management for service hosting platforms. In: Proceedings of the 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing. 2009, 1–8
Mehta H K, Kanungo P, Chandwani M. Performance enhancement of scheduling algorithms in clusters and grids using improved dynamic load balancing techniques. In: Proceedings of the 20th International Conference Companion on World Wide Web. 2011, 385–390
Chapman C, Emmerich W, Márquez F G, Clayman S, Galis A. Software architecture definition for on-demand cloud provisioning. Cluster Computing, 2012, 15(2): 79–100
Ardagna D, Panicucci B, Passacantando M. A game theoretic formulation of the service provisioning problem in cloud systems. In: Proceedings of the 20th International Conference Companion on World Wide Web (WWW). 2011, 177–186
Qiang L, Qin-Fen H, Li-Min X, Zhou-Jun L. Adaptive management and multi-objective optimization for virtual machine placement in cloud computing. Chinese Journal of Computers, 2011, 34(12): 2253–2264
Kaur P D, Chana I. A resource elasticity framework for QoS-aware execution of cloud applications. Future Generation Computer Systems, 2014, 37: 14–25
Son S, Jun S C. Negotiation-based flexible SLA establishment with SLA-driven resource allocation in cloud computing. In: Proceedings of the 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid). 2013, 168–171
García A G, Espert I B, García V H. SLA-driven dynamic cloud resource management. Future Generation Computer Systems, 2014, 31: 1–11
Wang L, Zhan J, Shi W, Liang Y, Yuan L. In cloud, do MTC or HTC service providers benefit from the economies of scale? In: Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers. 2009, 7
Jin H, Qin H, Wu S, Guo X. CCAP: a cache contention-aware virtual machine placement approach for HPC cloud. International Journal of Parallel Programming, 2015, 43(3): 403–420
Chen H, Wu S, Di S, Zhou B, Xie Z, Jin H, Shi X. Communicationdriven scheduling for virtual clusters in cloud. In: Proceedings of the 23rd International Symposium on High-performance Parallel and Distributed Computing (HPDC). 2014, 125–128
Wu S, Chen H, Di S, Zhou B, Xie Z, Jin H, Shi X. Synchronizationaware scheduling for virtual clusters in cloud. IEEE Transactions on Parallel and Distributed Systems, 2015, 26(10): 2890–2901
Eldred M S. Optimization strategies for complex engineering applications. Technical Report, Sandia National Labs., Albuquerque, NM (United States), 1998
Keahey K. Cloud computing for science. In: Proceedings of the 21st International Conference on Scientific and Statistical Database Management. 2009, 478
Deelman E, Singh G, Livny M, Berriman B, Good J. The cost of doing science on the cloud: the montage example. In: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (ICS). 2008, 50
Wang L, Tao J, Kunze M, Castellanos A C, Kramer D, Karl W. Scientific cloud computing: Early definition and experience. In: Proceedings of the IEEE International Conference on High Performance Computing and Communications. 2008, 825–830
Ludäscher B, Altintas I, Berkley C, Higgins D, Jaeger E, Jones M, Lee E A, Tao J, Zhao Y. Scientific workflow management and the Kepler system. Concurrency and Computation: Practice and Experience, 2006, 18(10): 1039–1065
Oinn T, Addis M, Ferris J, Marvin D, Senger M, Greenwood M, Carver T, Glover K, Pocock M R, Wipat A, Li P. Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics, 2004, 20(17): 3045–3054
Author information
Authors and Affiliations
Corresponding author
Additional information
Haibao Chen is currently working toward the PhD degree in Service Computing Technology and System Lab (SCTS) and Cluster and Grid Lab (CGCL) at Huazhong University of Science and Technology, China. He is also with the Big Data and Cloud Lab, School of Computer and information engineering, Chuzhou University, China. His research interests include parallel and distributed computing, virtualization, and resource scheduling on cloud computing.
Song Wu is a professor of computer science and engineering at Huazhong University of Science and Technology (HUST), China. He received his PhD from HUST in 2003. His current research interests include cloud computing, system virtualization, datacenter management, storage system, in-memory computing and so on.
Hai Jin received the PhD degree in computer engineering from Huazhong University of Science and Technology, China in 1994. He is the chief scientists of National 973 Basic Research Program Project of Virtualization Technology of Computing System, and Cloud Security. His research interests include computer architecture, virtualization technology, cluster computing and cloud computing, peerto- peer computing, network storage, and network security.
Wenguang Chen received the BS and PhD degrees in computer science from Tsinghua University, China in 1995 and 2000 respectively. He is a professor in Department of Computer Science and Technology, Tsinghua University. His research interest is in parallel and distributed computing, and programming model.
Jidong Zhai received his PhD degree from Tsinghua University, China in 2010. He is an assistant professor at the Department of Computer Science and Technology, Tsinghua University, China. His research focuses on high performance computing, especially performance analysis and optimization for large-scale parallel applications and performance evaluation for computer systems.
Yingwei Luo received his BS degree from Zhejiang University, China in 1993, and his MS and PhD degrees from Peking University, China in 1996 and 1999 respectively. He is currently a professor in Peking University, China. His research interests include virtualization technologies, distributed computing and geographic information system, etc.
Xiaolin Wang received his BS and PhD degrees from Peking University, China in 1996 and 2001 respectively. He is an associate professor in Peking University. His research interests include virtualization technologies, distributed computing and geographic information system, etc.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Chen, H., Wu, S., Jin, H. et al. A survey of cloud resource management for complex engineering applications. Front. Comput. Sci. 10, 447–461 (2016). https://doi.org/10.1007/s11704-015-4207-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11704-015-4207-x