skip to main content
10.1145/3655532.3655574acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicrsaConference Proceedingsconference-collections
research-article

Fuzzing for Deep Learning Models

Published: 28 June 2024 Publication History

Abstract

Deep learning models have been widely used in security fields such as autonomous vehicles, and the testing of their quality problems has gradually attracted attention. Fuzzing has become an important testing method because of its efficient fault revealing ability. The quality and effectiveness of test cases generated by existing fuzzing methods are not high. In this paper, we propose a fuzzing method for deep learning models that uses heuristic policy mutation inputs to generate test cases. Thus, the quality of test cases is improved, so that the classification error of the model can be tested faster. By testing the image classification model, the experiments show that the generated test cases improve the neuron coverage of the model, and the time to find the model classification error is shorter.

References

[1]
Zhang JM, Harman M, Ma L, Liu Y. Machine learning testing: Survey, landscapes and horizons. IEEE Trans. on Software Engineering,2022, 48(1): 1–36.
[2]
Sun YC, Huang XW, Kroening D, Sharp J, Hill M, Ashmore R. Testing deep neural networks. arXiv:1803.04792, 2018.
[3]
Huang XW, Kroening D, Ruan WJ, Sharp J, Sun YC, Thamo E, Wu M, Yi XP. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review, 2020, 37: 100270.
[4]
Eykholt K, Evtimov I, Fernandes E, Li B, Rahmati A, Xiao CW, Prakash A, Kohno T, Song D. Robust physical-world attacks on deeplearning visual classification. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City: IEEE,2018. 1625–1634.
[5]
Sekhon J, Fleming C. Towards improved testing for deep learning. In: Proc. of the 41st IEEE/ACM Int'l Conf. on Software Engineering(ICSE-NIER). Montreal: IEEE, 2019. 85–88.
[6]
Hongliang Liang and Xiaoxiao Pei and Xiaodong Jia and Wuwei Shen and Jian Zhang Fuzzing: State of the Art IEEE Transactions on Reliability (2018):67.
[7]
Gao X,Saha RK,Prasad MR,Roychoudhury A.Fuzz testing based data augmentation to improve robustness of deep neural networks. In: Proc. of the 42nd IEEE/ACM Int'l Conf. on Software Engineering (ICSE). Seoul: IEEE, 2020.1147–1158.
[8]
Xie XF, Ma L, Xu FJ, Xue MH, Chen HX, Liu Y, Zhao JJ, Li B, Yin JX, See S. DeepHunter: A coverage-guided fuzz testing framework for deep neural networks. In: Proc. of the 28th ACM SIGSOFT Int ’l Symp. on Software Testing and Analysis. Beijing: ACM, 2019.146–157.
[9]
Xie XF,Chen HX, Li Y,Ma L, Liu Y, Zhao JJ.Coverage-guided fuzzing for feedforward neural networks.In: Proc.of the 34th IEEE/ACM Int'l Conf.on AutomatedSoftware Engineering (ASE).San Diego:IEEE,2019.1162–1165.
[10]
Pei KX, Cao YZ, Yang JF, Jana S. DeepXplore: Automated whitebox testing of deep learning systems. In: Proc. of the 26th Symp. onOperating Systems Principles. Shanghai: ACM, 2017. 1–18.
[11]
Guo J, Jiang Y, Zhao Y, DLFuzz: Differential fuzzing testing of deep learning systems. In: Proc. of the 2018 26th ACM Joint Meeting on European Software Engineering Conf. and Symp. on the Foundations of Software Engineering. ACM, 2018. 739−743.
[12]
Odena A, Olsson C, Andersen D, Goodfellow I. Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. In: Proc. of the 36th Int'l Conf. on Machine Learning. Long Beach: ICML, 2019. 4901–4911.
[13]
Park LH, Oh S, Kim J, Chung S, Kwon T. Poster: Effective layers in coverage metrics for deep neural networks. In: Proc. of the 27th ACM SIGSAC Conf. on Computer and Communications Security. London: ACM, 2019. 2681–2683.
[14]
Ma L, Xu FJ, Zhang FY, Sun JY, Xue MH, Li B, Chen CY, Su T, Li L, Liu Y, Zhao JJ, Wang YD. DeepGauge: Multi-granularity testing criteria for deep learning systems. In: Proc. of the 33rd IEEE/ACM Int ’l Conf. on Automated Software Engineering (ASE).Montpellier: IEEE, 2018. 120–131.
[15]
Park LH, Oh S, Kim J, Chung S, Kwon T. Poster: Effective layers in coverage metrics for deep neural networks. In: Proc. of the 27th ACM SIGSAC Conf. on Computer and Communications Security. London: ACM, 2019. 2681–2683.
[16]
Deng J, Dong W, Socher R, Li LJ, Li K, Li FF. ImageNet: A large-scale hierarchical image database. In: Proc. of the 2009 Conf. On
[17]
Computer Vision and Pattern Recognition. Miami: IEEE, 2009. 248–255.
[18]
LeCun Y, Cortes C, Burges C. MNIST handwritten digit database. AT&T Labs. 2010.
[19]
Papadakis M, Kintis M, Zhang J, Jia Y, Le Traon Y, Harman M. Mutation testing advances: An analysis and survey. Advances inComputers, 2019, 112: 275–378.
[20]
Shen WJ, Wan J, Chen ZY. MuNN: Mutation analysis of neural networks. In: Proc. of the 18th IEEE Int'l Conf. on Software Quality,Reliability and Security Companion (QRS-C). Lisbon: IEEE, 2018. 108–115.
[21]
Zhang Y, Chen Y, Cheung S C, An empirical study on TensorFlow program bugs[C]//Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2018: 129-140.
[22]
Hu Q, Ma L, Xie XF, Yu B, Liu Y, Zhao JJ. DeepMutation++: A mutation testing framework for deep learning systems. In: Proc. of the 34th IEEE/ACM Int'l Conf. on Automated Software Engineering (ASE). San Diego: IEEE, 2019. 1158–1161.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICRSA '23: Proceedings of the 2023 6th International Conference on Robot Systems and Applications
September 2023
335 pages
ISBN:9798400708039
DOI:10.1145/3655532
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 June 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Fuzzing
  2. Model testing
  3. Mutation

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICRSA 2023

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 31
    Total Downloads
  • Downloads (Last 12 months)31
  • Downloads (Last 6 weeks)3
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media