research-article

Heterogeneous Computing and Applications in Deep Learning: A Survey

Authors:

Mingqing ZhangAuthors Info & Claims

CSSE '22: Proceedings of the 5th International Conference on Computer Science and Software Engineering

Pages 383 - 387

https://doi.org/10.1145/3569966.3570075

Published: 20 December 2022 Publication History

Abstract

With the rapid development of deep learning, a variety of neural network models emerge in endlessly, which leads to a huge demand for computing resources. For the intensive numerical computation of neural networks, various computing devices represented by GPUs are favored by researchers. Heterogeneous computing is a kind of technology that can integrate a variety of computing devices with different architectures, and it will be further developed. Therefore, this paper reviews research on some key technologies of heterogeneous computing, including the architecture of heterogeneous computing, the programming language of heterogeneous computing, and the scheduling algorithm for heterogeneous systems. Then, we focus on the research of heterogeneous computing in deep learning, including the parallel technology of neural networks and optimization technology based on heterogeneous systems. Finally, the present research situation is discussed and analyzed, and the future research direction is prospected, aiming to provide some basis for related research.

References

[1]

Deng J, Dong W, Socher R, ImageNet: A large-scale hierarchical image database[C]//2009 IEEE conference on computer vision and pattern recognition. IEEE, 2009: 248-255.

[2]

Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks[J]. science, 2006, 313(5786): 504-507.

[3]

Wu J, Qi X, Gao Y. Review of Programming Models for Heterogeneous Parallel Computing[J]. AEROSPACE SHANGHAI (CHINESE & ENGLISH),2021,38(04):1-11. https://doi.org/10.19328/j.cnki.2096-8655.2021.04.001

[4]

Yang W, Wang H, Zhang Y, Lin S, Cai Q. Survey of Heterogeneous Hybrid Parallel Computing[J]. Computer Science,2020,47(08):5-16+3. https://doi.org/10.11896/jsjkx.200600045

[5]

KHRONOS Group. The open standard for parallel programming of heterogeneous systems. 2022. http://www.khronos.org/opencl

[6]

NVIDIA CUDA toolkit. 2022. https://developer.nvidia.com/cuda-downloads

[7]

OpenAcc: Directives for accelerators. 2022. http://www.openacc.org/

[8]

Ueng S Z, Lathara M, Baghsorkhi S S, CUDA-lite: Reducing GPU programming complexity[C]//International Workshop on Languages and Compilers for Parallel Computing. Springer, Berlin, Heidelberg, 2008: 1-15.

[9]

Han T D, Abdelrahman T S. hiCUDA: High-level GPGPU programming[J]. IEEE Transactions on Parallel and Distributed Systems, 2010, 22(1): 78-90.

Digital Library

[10]

Tang T. Research on programming model and compiler optimizations for CPU-GPU heterogeneous parallel systems [Ph.D. Thesis]. Changsha: Graduate School of National University of Defense Technology, 2011. https://www.dissertationtopic.net/doc/1475908

[11]

Buck I, Foley T, Horn D, Brook for GPUs: stream computing on graphics hardware[J]. ACM transactions on graphics (TOG), 2004, 23(3): 777-786.

Digital Library

[12]

Gregory K, Miller A. C++ AMP: accelerated massive parallelism with Microsoft Visual C++[M]. Microsoft Press, 2012.

[13]

Linderman M D, Collins J D, Wang H, Merge: a programming model for heterogeneous multi-core systems[J]. ACM SIGOPS operating systems review, 2008, 42(2): 287-296.

[14]

Auerbach J, Bacon D F, Cheng P, Lime: a java-compatible and synthesizable language for heterogeneous architectures[C]//Proceedings of the ACM international conference on Object oriented programming systems languages and applications. 2010: 89-108.

[15]

Garg R, Amaral J N. Compiling python to a hybrid execution environment[C]//Proceedings of the 3rd Workshop on General-Purpose Computation on Graphics Processing Units. 2010: 19-30.

[16]

Catanzaro B, Garland M, Keutzer K. Copperhead: compiling an embedded data parallel language[C]//Proceedings of the 16th ACM symposium on Principles and practice of parallel programming. 2011: 47-56.

[17]

Asanovic K, Bodik R, Catanzaro B C, The landscape of parallel computing research: A view from berkeley[J]. 2006.

[18]

Sulaiman M, Halim Z, Waqas M, A hybrid list-based task scheduling scheme for heterogeneous computing[J]. The Journal of Supercomputing, 2021: 1-37.

[19]

Ahmad W, Alam B. An efficient list scheduling algorithm with task duplication for scientific big data workflow in heterogeneous computing environments[J]. Concurrency and Computation: Practice and Experience, 2021, 33(5): e5987.

[20]

Djigal H, Feng J, Lu J, IPPTS: an efficient algorithm for scientific workflow scheduling in heterogeneous computing systems[J]. IEEE Transactions on Parallel and Distributed Systems, 2020, 32(5): 1057-1071.

Digital Library

[21]

Liang L, Zhang Q, Song P, Overlapping communication and computation of GPU/CPU heterogeneous parallel spatial domain decomposition MOC method[J]. Annals of Nuclear Energy, 2020, 135: 106988.

[22]

Zhang J, Zhan J, Li J, Optimizing execution for pipelined‐based distributed deep learning in a heterogeneously networked GPU cluster[J]. Concurrency and Computation: Practice and Experience, 2020, 32(23): e5923.

[23]

Nidaw B Y, Oh M H, Kim Y W. Appropriate Synchronization Time Allocation for Distributed Heterogeneous Parallel Computing Systems[J]. KSII Transactions on Internet and Information Systems (TIIS), 2019, 13(11): 5446-5463.

[24]

Andronikos T, Ciorba F M, Riakiotakis I, Studying the impact of synchronization frequency on scheduling tasks with dependencies in heterogeneous systems[J]. Performance Evaluation, 2010, 67(12): 1324-1339.

Digital Library

[25]

Zhu H, Li P, Jiao L, Yang S, Hou B. Review of parallel Deep Neural Network[J]. Chinese Journal of Computers,2018,41(08):1861-1881. https://doi.org/10.7544/issn1000-1239.2021.20210166

[26]

Raina R, Madhavan A, Ng A Y. Large-scale deep unsupervised learning using graphics processors[C]//Proceedings of the 26th annual international conference on machine learning. 2009: 873-880.

[27]

Yadan O, Adams K, Taigman Y, Multi-GPU training of convnets[J]. arXiv preprint arXiv:1312.5853, 2013.

[28]

Bottleson J, Kim S Y, Andrews J, clCaffe: Opencl accelerated caffe for convolutional neural networks[C]//2016 IEEE international parallel and distributed processing symposium workshops (IPDPSW). IEEE, 2016: 50-57.

[29]

Viebke A, Memeti S, Pllana S, CHAOS: a parallelization scheme for training convolutional neural networks on Intel Xeon Phi[J]. The Journal of Supercomputing, 2019, 75(1): 197-227.

Digital Library

[30]

Suda N, Chandra V, Dasika G, Throughput-optimized OpenCL-based FPGA accelerator for large-scale convolutional neural networks[C]//Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. 2016: 16-25.

[31]

Chen T, Du Z, Sun N, DianNao: A small-footprint high-throughput accelerator for ubiquitous machine-learning[J]. ACM SIGARCH Computer Architecture News, 2014, 42(1): 269-284.

Digital Library

[32]

Song K, Liu Y, Wang R, Restricted Boltzmann machines and deep belief networks on sunway cluster[C]//2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, 2016: 245-252.

[33]

Sarkar S, Alavani G. How Easy it is to Write Software for Heterogeneous Systems?[J]. ACM SIGSOFT Software Engineering Notes, 2018, 42(4): 1-7.

Digital Library

[34]

Liu Y, Lü F, Wang L, Chen L, Cui HM, Feng XB. Research on heterogeneous parallel programming model[J]. Journal of Software, 2014,25(7):1459-1475. https://doi.org/10.13328/j.cnki.jos.004608

Cited By

Callejas SLira HBerry AMartí LSanchez-Pi N(2025)No Plankton Left Behind: Preliminary Results on Massive Plankton Image RecognitionHigh Performance Computing10.1007/978-3-031-80084-9_12(170-185)Online publication date: 14-Feb-2025
https://doi.org/10.1007/978-3-031-80084-9_12
Yin JWu FSu H(2024)An Image-Retrieval Method Based on Cross-Hardware Platform FeaturesApplied System Innovation10.3390/asi70400647:4(64)Online publication date: 23-Jul-2024
https://doi.org/10.3390/asi7040064
Wang GGu CLi JWang JChen XZhang H(2023)Heterogeneous Flight Management System (FMS) Design for Unmanned Aerial Vehicles (UAVs): Current Stages, Challenges, and OpportunitiesDrones10.3390/drones70603807:6(380)Online publication date: 6-Jun-2023
https://doi.org/10.3390/drones7060380

Index Terms

Heterogeneous Computing and Applications in Deep Learning: A Survey
1. General and reference
  1. Document types
    1. Surveys and overviews

Recommendations

Algorithm-system scalability of heterogeneous computing

Scalability is a key factor of the design of distributed systems and parallel algorithms and machines. However, conventional scalabilities are designed for homogeneous parallel processing. There is no suitable and commonly accepted definition of ...
Heterogeneous Computing for Undergraduates: Introducing the ToUCH Module Repository
SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 2

The need for increased performance per watt, coupled with the demands of processing diverse workloads, has triggered an industry shift towards heterogeneous computing systems. Integration of high-performance CPUs with energy-efficient GPUs is now common ...
A Survey Paper on Task Scheduling Methods in Cluster Computing Environment for High Performance
ACCT '15: Proceedings of the 2015 Fifth International Conference on Advanced Computing & Communication Technologies

Parallel computing perform concurrently execution tasks on distributed nodes. Large application split up into tasks and run on number on nodes for high performance computing. Cluster environment composed with heterogeneous devices and software ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

CSSE '22: Proceedings of the 5th International Conference on Computer Science and Software Engineering

October 2022

753 pages

ISBN:9781450397780

DOI:10.1145/3569966

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 December 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CSSE 2022

CSSE 2022: 2022 5th International Conference on Computer Science and Software Engineering

October 21 - 23, 2022

Guilin, China

Acceptance Rates

Overall Acceptance Rate 33 of 74 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
269
Total Downloads

Downloads (Last 12 months)102
Downloads (Last 6 weeks)16

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Callejas SLira HBerry AMartí LSanchez-Pi N(2025)No Plankton Left Behind: Preliminary Results on Massive Plankton Image RecognitionHigh Performance Computing10.1007/978-3-031-80084-9_12(170-185)Online publication date: 14-Feb-2025
https://doi.org/10.1007/978-3-031-80084-9_12
Yin JWu FSu H(2024)An Image-Retrieval Method Based on Cross-Hardware Platform FeaturesApplied System Innovation10.3390/asi70400647:4(64)Online publication date: 23-Jul-2024
https://doi.org/10.3390/asi7040064
Wang GGu CLi JWang JChen XZhang H(2023)Heterogeneous Flight Management System (FMS) Design for Unmanned Aerial Vehicles (UAVs): Current Stages, Challenges, and OpportunitiesDrones10.3390/drones70603807:6(380)Online publication date: 6-Jun-2023
https://doi.org/10.3390/drones7060380

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten