Cited By
View all- Zhang XJiang WShen CLi QWang QLin CGuan X(2025)Deep Learning Library Testing: Definition, Methods and ChallengesACM Computing Surveys10.1145/371649757:7(1-37)Online publication date: 5-Feb-2025
Research on Deep Neural Network Testing Techniques
Pages 113 - 119
Abstract
Profiting by the rapid development of computer science and technology, deep neural networks have been widely used in security-related fields such as face recognition, automatic driving, medical diagnosis and decision-making reasoning, and there is an urgent need for testers to conduct comprehensive and in-depth testing of these software to ensure their quality and security. However, intelligent software based on neural networks is fundamentally different from traditional software. In recent years, more and more researchers have shifted their attention from traditional software testing to intelligent software testing, and a series of evaluation criteria, test frameworks, and test case generation methods, etc. have been proposed for deep neural network models. This paper summarises and concludes the existing research from the perspectives of testing techniques based on test adequacy theory, testing techniques based on traditional testing theory and testing techniques based on adversarial samples. Finally, it summarises and looks forward to deep neural network testing and points out the problems in deep neural network testing, in order to provide some thoughts for researchers in related fields.
References
[1]
He K, Zhang X, Ren S, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[C]//Proceedings of the IEEE international conference on computer vision. 2015, 1026-1034.
[2]
Szegedy C, Zaremba W, Sutskever I, Intriguing properties of neural networks [J]. arXiv preprint arXiv:1312.6199, 2013.
[3]
Ingle S, Phute M. Tesla autopilot: semi autonomous driving, an uptick for future autonomy [J]. International Research Journal of Engineering and Technology, 2016, 3(9): 369-372.
[4]
Lab T K S. Experimental security research of Tesla autopilot[J]. Tencent Keen Security Lab, 2019.
[5]
Glenford J. Myers, Tom Badgett, Corey Sandler. The art of software testing[M]. Feng Wang, Jie Chen, translation. Beijing, 2006.
[6]
Jinhui Shan, Ying Jang, Ping Sun. Research progress of software testing[J]. Journal of Peking University (Natural Science), 2005, (1): 134-145.
[7]
Aditya P. Mathur. Software testing basics tutorial[M]. Feng Wang, Changguo Guo, translation Beijing: China Machine Press, 2011.
[8]
Kexin Pei, Yinzhi Cao, Junfeng Yang, Deepxplore: Automated whitebox testing of deep learning systems[C]//proceedings of the 26th Symposium on Operating Systems Principles. 2017, 1-18.
[9]
Sun Y, Huang X, Kroening D, Testing deep neural networks [J]. arXiv preprint arXiv:1803.04792, 2018.
[10]
Ma L, Juefei-Xu F, Zhang F, Deepgauge: Multi-granularity testing criteria for deep learning systems[C]//Proceedings of the 33rd ACM/IEEE international conference on automated software engineering. 2018, 120-131.
[11]
Deng L. The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] [J]. IEEE Signal Processing Magazine, 2012, 29(6): 141-142.
[12]
Deng J, Dong W, Socher R, ImageNet: A Large-Scale Hierarchical Image Database [J]. Proc. CVPR, 2009, 2009.
[13]
Lecun Y, Bottou L. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
[14]
He K, Zhang X, Ren S, Deep Residual Learning for Image Recognition [J]. IEEE, 2016.
[15]
Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition [J]. Computer Science, 2014.
[16]
Goodfellow I J, Shlens J, Szegedy C. Explaining and harnessing adversarial examples [J]. arXiv preprint arXiv: 1412.6572, 2014.
[17]
Carlini N, Wagner D. Towards Evaluating the Robustness of Neural Networks[C]//2017 IEEE Symposium on Security and Privacy (SP). 2017.
[18]
Papernot N, Mcdaniel P, Jha S, The Limitations of Deep Learning in Adversarial Settings [J]. IEEE, 2015.
[19]
Kurakin A, Goodfellow I J, Bengio S. Adversarial examples in the physical world[M]//Artificial intelligence safety and security. Chapman and Hall/CRC, 2018, 99-112.
[20]
Ma L, Juefei-Xu F, Xue M, DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems[C]//2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). 2019.
[21]
Kim J, Feldt R, Yoo S. Guiding Deep Learning System Testing Using Surprise Adequacy [J]. ACM, 2019.
[22]
Sekhon J, Fleming C. Towards Improved Testing For Deep Learning[C]//2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). 2019.
[23]
Lee S, Cha S, Lee D, Effective white-box testing of deep neural networks with adaptive neuron-selection strategy[C]//ISSTA ’20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2020.
[24]
Wang D, Wang Z, Fang C, DeepPath: Path-Driven Testing Criteria for Deep Neural Networks[C]//2019 IEEE International Conference On Artificial Intelligence Testing (AITest). 2019.
[25]
Du X, Xie X, Li Y, DeepCruiser: Automated Guided Testing for Stateful Deep Learning Systems[J]. 2018.
[26]
Gerasimou S, Eniser H F, Sen A, Importance-driven deep learning system testing[C]//Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 2020, 702-713.
[27]
Ma L, Zhang F, Sun J, DeepMutation: Mutation Testing of Deep Learning Systems[J]. IEEE, 2018.
[28]
Shen W, Wan J, Chen Z. MuNN: Mutation Analysis of Neural Networks[C]//2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C). 2018, 108-115.
[29]
Klampfl L, Chetouane N, Wotawa F. Mutation Testing for Artificial Neural Networks: An Empirical Evaluation[C]//2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS). 2020, 356-365.
[30]
Tambon F, Khomh F, Antoniol G. A probabilistic framework for mutation testing in deep neural networks[J]. Information and Software Technology, 2023, 155: 107129.
[31]
Guo J, Jiang Y, Zhao Y, Dlfuzz: Differential fuzzing testing of deep learning systems[C]//Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2018, 739-743.
[32]
Odena A, Olsson C, Andersen D, Tensorfuzz: Debugging neural networks with coverage-guided fuzzing[C]//International Conference on Machine Learning. PMLR, 2019, 4901-4911.
[33]
Tao C, Tao Y, Guo H, DLRegion: Coverage-guided fuzz testing of deep neural networks with region-based neuron selection strategies[J]. Information and Software Technology, 2023, 162: 107266.
[34]
Chen T Y. Metamorphic testing: A simple method for alleviating the test oracle problem[C]//2015 IEEE/ACM 10th International Workshop on Automation of Software Test. IEEE, 2015, 53-54.
[35]
Liu Jialuo, Yao Yi, Huang Song, Metamorphosis testing Framework for machine Learning image classification programs [J]. Computer Engineering and Applications, 2020, 56(17): 69-77.
[36]
Chen T Y, Cheung S C, Yiu S M. Metamorphic testing: a new approach for generating next test cases[J]. arXiv preprint arXiv:2002.12543, 2020.
[37]
Zhang M, Zhang Y, Zhang L, DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems[C]//IEEE/ACM International Conference on Automated Software Engineering. 2018.
[38]
Tian Y, Pei K, Jana S, Deeptest: Automated testing of deep-neural-network-driven autonomous cars[C]//Proceedings of the 40th international conference on software engineering. 2018: 303-314.
[39]
Zhang Z, Wang P, Guo H, DeepBackground: Metamorphic testing for Deep-Learning-driven image recognition systems accompanied by Background-Relevance[J]. Information and Software Technology, 2021, 140: 106701.
[40]
Xiao D, LIU Z, Yuan Y, Metamorphic Testing of Deep Learning Compilers[J]. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2022, 6(1): 1-28.
[41]
Chandrasekaran J, Lei Y, Kacker R, A combinatorial approach to testing deep neural network-based autonomous driving systems[C]//2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2021: 57-66.
[42]
Chandrasekaran J, Patel A R, Lei Y, Evaluation of T-Way Testing of DNNs in Autonomous Driving Systems[C]//2021 IEEE International Conference on Artificial Intelligence Testing (AITest). IEEE, 2021: 17-18.
[43]
Kitamura T, Zhao Z, Toda T. Applying Combinatorial Testing to Verification-Based Fairness Testing[C]//PAPADAKIS M, VERGILIO S R. Search-Based Software Engineering. Cham: Springer International Publishing, 2022: 101-107.
Index Terms
- Research on Deep Neural Network Testing Techniques
Recommendations
Concolic testing for deep neural networks
ASE '18: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software EngineeringConcolic testing combines program execution and symbolic analysis to explore the execution paths of a software program. In this paper, we develop the first concolic testing approach for Deep Neural Networks (DNNs). More specifically, we utilise ...
Testing deep neural networks (keynote)
SPLASH Companion 2020: Companion Proceedings of the 2020 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for HumanityThe reliability of software that has a Deep Neural Network (DNN) as a component is urgently important today given the increasing number of critical applications being deployed with DNNs. The need for reliability raises a need for rigorous testing of the ...
Neuron importance-aware coverage analysis for deep neural network testing
AbstractDeep Neural Network (DNN) models are widely used in many cutting-edge domains, such as medical diagnostics and autonomous driving. However, an urgent need to test DNN models thoroughly has increasingly risen. Recent research proposes various ...
Comments
Information & Contributors
Information
Published In
October 2023
1065 pages
ISBN:9798400709449
DOI:10.1145/3650215
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 16 April 2024
Check for updates
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICMLCA 2023
ICMLCA 2023: 2023 4th International Conference on Machine Learning and Computer Application
October 27 - 29, 2023
Hangzhou, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 52Total Downloads
- Downloads (Last 12 months)52
- Downloads (Last 6 weeks)11
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
Cited By
View all- Zhang XJiang WShen CLi QWang QLin CGuan X(2025)Deep Learning Library Testing: Definition, Methods and ChallengesACM Computing Surveys10.1145/371649757:7(1-37)Online publication date: 5-Feb-2025
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format