research-article

Performance Evaluation and Analysis of Deep Learning Frameworks

Authors:

Hao XuAuthors Info & Claims

AIPR '22: Proceedings of the 2022 5th International Conference on Artificial Intelligence and Pattern Recognition

Pages 38 - 44

https://doi.org/10.1145/3573942.3573948

Published: 16 May 2023 Publication History

Abstract

The rapid development of deep learning has contributed to the increasing number of open-source deep learning frameworks, and in practice, benchmarking deep learning frameworks to effectively understand the performance characteristics of these frameworks and make choices becomes a challenge. Based on this, this paper uses three types of neural networks (convolutional neural networks, recurrent neural networks, and vision transformer models) to conduct extensive experimental evaluation and analysis of three popular deep learning frameworks, TensorFlow, PyTorch, and PaddlePaddle. Experiments are mainly conducted in CPU and GPU environments using different datasets, and performance parameters such as accuracy, training time, inference time, hardware utilization and other non-performance factors are considered. Finally, the performance characteristics, advantages and disadvantages of different frameworks are analyzed based on the above indexes, which provides theoretical guidance for users to choose.

References

[1]

LeCun Y, Bengio Y, Hinton G. Deep learning[J]. Nature, 2015, 521(7553): 436-444

[2]

Abadi M, Barham P, Chen J, TensorFlow: A system for large-scale machine learning[J]. Usenix Association, 2016: 265-283.

[3]

Jia Y, Shelhamer E, Donahue J, Caffe: Convolutional architecture for fast feature embedding[J].ACM, 2014: 675-678.

[4]

Paszke A, Gross S, Massa F, Pytorch: An imperative style, high-performance deep learning library[J]. Advances in neural information processing systems, 2019: 32

[5]

Bastien F, Lamblin P, Pascanu R, Theano: new features and speed improvements[J]. 2012

[6]

Paddle paddle:Https://www.paddlepaddle.org.cn/

[7]

Coleman C, Narayanan D, Kang D, Dawnbench: An end-to-end deep learning benchmark and competition [J]. Training, 2017, 100(101): 102.

[8]

Shams S Y., Platania, R., Lee, K, Evaluation of deep learning frameworks over different HPC architectures[C]//International Conference on Distributed Computing Systems, Atlanta, Georgia 2017, 1389–1396

[9]

Wang Q C, Guo G D. Benchmarking deep learning techniques for face recognition[J].Visual Communication and Image Representation,2019, 65(C) : 102663-102663

[10]

Bahrampour S, Ramakrishnan N, Schott L, : Comparative study of caffe, neon, theano, and torch for deep learning[C] //International Conference on Learning Representations,San Juan, Puerto Rico, 2016,1-11

[11]

Wu Y Z, Liu L., Pu C T, A comparative measurement study of deep learning as a service framework[J].IEEE Transactions on Services Computing,2022,15(1):551-566

[12]

Shi S H., Wang Q., Xu P F, Benchmarking state-of-the-art deep learning software tools[C]//International Conference on Cloud Computing and Big Data, Taipa, Macau, China.2016,99-104

[13]

Chen T Q, Li M, Li Y T, Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. (2015)

[14]

Bahrampour S, Ramakrishnan N, Schott L, Comparative Study of Deep Learning Software Frameworks[J]. Computer Science, 2016

[15]

Elshawi R,Wahab A, Barnawi A, DLBench : a comprehensive experimental evaluation of deep learning frameworks[J]. Cluster Computing, 2021,24(3),2017-2038

Digital Library

[16]

Pothos V, Vassalos E, Theodorakopoulos I, Deep Learning Inference with Dynamic Graphs on Heterogeneous Platforms[J]. International Journal of Parallel Programming, 2021, 49(2): 158-176.

[17]

Li Z W, Liu F, Yang W J, A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects[J]. IEEE Transactions on Neural Networks And Learning System,2021,PP

Cited By

Xie XWang HJian ZLi TWang WXu ZWang G(2024)Memory-Efficient and Secure DNN Inference on TrustZone-enabled Consumer IoT DevicesIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621088(2009-2018)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621088
Tian JWei Y(2024)An Inference Performance Evaluation of TensorFlow and PyTorch on GPU Platform Using Image Super-Resolution Workloads2024 16th International Conference on Electronics, Computers and Artificial Intelligence (ECAI)10.1109/ECAI61503.2024.10607564(1-6)Online publication date: 27-Jun-2024
https://doi.org/10.1109/ECAI61503.2024.10607564
Shi MZhang ZZhang WMa YLi HLee E(2024)The study of self-organised behaviours and movement pattern of pedestrians during fire evacuations: virtual experiments and surveySafety Science10.1016/j.ssci.2023.106373170(106373)Online publication date: Feb-2024
https://doi.org/10.1016/j.ssci.2023.106373
Show More Cited By

Index Terms

Performance Evaluation and Analysis of Deep Learning Frameworks
1. Networks
  1. Network performance evaluation
    1. Network performance analysis

Recommendations

Parametric Learning of Deep Convolutional Neural Network
IDEAS '15: Proceedings of the 19th International Database Engineering & Applications Symposium

Deep neural networks have recently been showing great potential on visual recognition tasks. However, it is also considered difficult to tune its parameters, and it has high training cost. This work focuses on analysis of several learning methods and ...
Hybrid deep learning cost evaluation using CNN with ANN for the plastic injection industry
Abstract
The plastic injection moulding industry has been growing and expanding rapidly. Cost evaluation is important to business operations and numerous aspects influence part pricing. However, the intricacy of production data often includes distinctive ...
DLBench: a comprehensive experimental evaluation of deep learning frameworks
Abstract
Deep Learning (DL) has achieved remarkable progress over the last decade on various tasks such as image recognition, speech recognition, and natural language processing. In general, three main crucial aspects fueled this progress: the increasing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AIPR '22: Proceedings of the 2022 5th International Conference on Artificial Intelligence and Pattern Recognition

September 2022

1221 pages

ISBN:9781450396899

DOI:10.1145/3573942

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 May 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

AIPR 2022

AIPR 2022: 2022 5th International Conference on Artificial Intelligence and Pattern Recognition

September 23 - 25, 2022

Xiamen, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
107
Total Downloads

Downloads (Last 12 months)48
Downloads (Last 6 weeks)4

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xie XWang HJian ZLi TWang WXu ZWang G(2024)Memory-Efficient and Secure DNN Inference on TrustZone-enabled Consumer IoT DevicesIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621088(2009-2018)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621088
Tian JWei Y(2024)An Inference Performance Evaluation of TensorFlow and PyTorch on GPU Platform Using Image Super-Resolution Workloads2024 16th International Conference on Electronics, Computers and Artificial Intelligence (ECAI)10.1109/ECAI61503.2024.10607564(1-6)Online publication date: 27-Jun-2024
https://doi.org/10.1109/ECAI61503.2024.10607564
Shi MZhang ZZhang WMa YLi HLee E(2024)The study of self-organised behaviours and movement pattern of pedestrians during fire evacuations: virtual experiments and surveySafety Science10.1016/j.ssci.2023.106373170(106373)Online publication date: Feb-2024
https://doi.org/10.1016/j.ssci.2023.106373
Wang BYang CZhu RLiu XZhou MZheng N(2023)Analysis of Performance and Optimization in MindSpore on Ascend NPUs2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS60453.2023.00237(1701-1708)Online publication date: 17-Dec-2023
https://doi.org/10.1109/ICPADS60453.2023.00237
Lu ZDu CJiang YXie XLi TYang F(2023)Quantitative evaluation of deep learning frameworks in heterogeneous computing environmentCCF Transactions on High Performance Computing10.1007/s42514-023-00168-66:1(94-111)Online publication date: 8-Sep-2023
https://doi.org/10.1007/s42514-023-00168-6

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten