Abstract
DNN testing is a critical way to guarantee the quality of DNNs. To obtain test oracle information, DNN testing requires a huge cost to label test inputs, which greatly affects the efficiency of DNN testing. To alleviate the labeling cost problem, the paper applies the idea of the spectrum-based fault location technique to DNN testing and proposes a novel test input prioritization approach for DNNs based on fault localization (called TPFL). TPFL first performs dynamic spectrum analysis on each neuron in the DNN. TPFL then proposes a suspiciousness measure that uses the neuron spectrum to identify suspicious neurons that cause the DNN to make wrong decisions. Finally, TPFL is based on the following key insight: a test input makes the suspicious neurons fully active, it indicates that this may be a bug-revealing input, so the input should have a higher priority. To evaluate, we conduct an empirical study on 3 widely used datasets and corresponding 8 DNN models. The experimental results show that TPFL performs well in both classification and regression models and overall outperforms most existing test input prioritization techniques.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abreu, R., Zoeteweij, P., Golsteijn, R.: A practical evaluation of spectrum-based fault localization. J. Syst. Softw. 82(11), 1780ā1792 (2009)
Aggarwal, A., Lohia, P., Nagar, S.: Black box fairness testing of machine learning models. In: Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, 26ā30 August 2019, pp. 625ā635. ACM (2019)
Bengio, Y., Mesnil, G., Dauphin, Y.N.: Better mixing via deep representations. In: Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16ā21 June 2013. JMLR Workshop and Conference Proceedings, vol. 28, pp. 552ā560. JMLR.org (2013)
Byun, T., Sharma, V., Vijayakumar, A.: Input prioritization for testing neural networks. In: IEEE International Conference On Artificial Intelligence Testing, AITest 2019, Newark, CA, USA, 4ā9 April 2019, pp. 63ā70. IEEE (2019)
Carlini, N., Wagner, D.A.: Towards evaluating the robustness of neural networks. In: 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, 22ā26 May 2017, pp. 39ā57. IEEE Computer Society (2017)
Chen, Z., Huang, X.: End-to-end learning for lane keeping of self-driving cars. In: IEEE Intelligent Vehicles Symposium, IV 2017, Los Angeles, CA, USA, 11ā14 June 2017, pp. 1856ā1860. IEEE (2017)
Cheng, D., Cao, C., Xu, C.: Manifesting bugs in machine learning code: an explorative study with mutation testing. In: 2018 IEEE International Conference on Software Quality, Reliability and Security, QRS 2018, Lisbon, Portugal, 16ā20 July 2018, pp. 313ā324. IEEE (2018)
Eniser, H.F., Gerasimou, S., Sen, A.: DeepFault: fault localization for deep neural networks. In: HƤhnle, R., van der Aalst, W. (eds.) FASE 2019. LNCS, vol. 11424, pp. 171ā191. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16722-6_10
Feng, Y., Shi, Q., Gao, X.: DeepGini: prioritizing massive tests to enhance the robustness of deep neural networks. In: ISSTA 2020: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, 18ā22 July 2020, pp. 177ā188. ACM (2020)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7ā9 May 2015, Conference Track Proceedings (2015)
Guo, J., Jiang, Y., Zhao, Y.: DLFuzz: differential fuzzing testing of deep learning systems. In: Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, 04ā09 November 2018, pp. 739ā743. ACM (2018)
He, K., Zhang, X., Ren, S.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27ā30 June 2016, pp. 770ā778. IEEE Computer Society (2016)
Khope, S.R., Elias, S.: Critical correlation of predictors for an efficient risk prediction framework of ICU patient using correlation and transformation of MIMIC-III dataset. Data Sci. Eng. 7(1), 71ā86 (2022)
Kim, J., Feldt, R., Yoo, S.: Guiding deep learning system testing using surprise adequacy. In: Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, 25ā31 May 2019, pp. 1039ā1049. IEEE/ACM (2019)
Kurakin, A., Goodfellow, I.J.: Adversarial examples in the physical world. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24ā26 April 2017, Workshop Track Proceedings. OpenReview.net (2017)
Lee, S., Cha, S., Lee, D.: Effective white-box testing of deep neural networks with adaptive neuron-selection strategy. In: ISSTA 2020: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, 18ā22 July 2020, pp. 165ā176. ACM (2020)
Liu, W., Wang, Z., Liu, X.: A survey of deep neural network architectures and their applications. Neurocomputing 234, 11ā26 (2017)
Ma, L., Juefei-Xu, F., Zhang, F.: DeepQauge: multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, 3ā7 September 2018, pp. 120ā131. ACM (2018)
Papernot, N., McDaniel, P.D., Jha, S.: The limitations of deep learning in adversarial settings. In: IEEE European Symposium on Security and Privacy, EuroS &P 2016, SaarbrĆ¼cken, Germany, 21ā24 March 2016, pp. 372ā387. IEEE (2016)
Pearson, S., Campos, J., Just, R.: Evaluating and improving fault localization. In: Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, 20ā28 May 2017, pp. 609ā620. IEEE/ACM (2017)
Pei, K., Cao, Y., Yang, J.: DeepXplore: automated whitebox testing of deep learning systems. In: Proceedings of the 26th Symposium on Operating Systems Principles, Shanghai, China, 28ā31 October 2017, pp. 1ā18. ACM (2017)
Rothermel, G., Untch, R.H., Chu, C.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929ā948 (2001)
Sun, Y., Huang, X., Kroening, D.: Testing deep neural networks. CoRR abs/1803.04792 (2018)
Sun, Y., Wu, M.: Concolic testing for deep neural networks. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, 3ā7 September 2018, pp. 109ā119. ACM (2018)
Tian, Y., Pei, K., Jana, S.: DeepTest: automated testing of deep-neural-network-driven autonomous cars. In: Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27āJune 03 2018, pp. 303ā314. ACM (2018)
Wang, Z., You, H., Chen, J.: Prioritizing test inputs for deep neural networks via mutation analysis. In: 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22ā30 May 2021, pp. 397ā409. IEEE (2021)
Wong, W.E., Gao, R., Li, Y.: A survey on software fault localization. IEEE Trans. Softw. Eng. 42(8), 707ā740 (2016)
Xie, X., Ma, L., Juefei-Xu, F.: DeepHunter: a coverage-guided fuzz testing framework for deep neural networks. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, 15ā19 July 2019, pp. 146ā157. ACM (2019)
Yan, R., Chen, Y., Gao, H.: Test case prioritization with neuron valuation based pattern. Sci. Comput. Program. 215, 102761 (2022)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818ā833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53
Zhang, J.M., Harman, M., Ma, L.: Machine learning testing: survey, landscapes and horizons. IEEE Trans. Softw. Eng. 48(2), 1ā36 (2022)
Zhang, K., Zhang, Y., Zhang, L.: Neuron activation frequency based test case prioritization. In: International Symposium on Theoretical Aspects of Software Engineering, TASE 2020, Hangzhou, China, 11ā13 December 2020, pp. 81ā88. IEEE (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
Ā© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tao, Y., Tao, C., Guo, H., Li, B. (2022). TPFL: Test Input Prioritization for Deep Neural Networks Based on Fault Localization. In: Chen, W., Yao, L., Cai, T., Pan, S., Shen, T., Li, X. (eds) Advanced Data Mining and Applications. ADMA 2022. Lecture Notes in Computer Science(), vol 13725. Springer, Cham. https://doi.org/10.1007/978-3-031-22064-7_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-22064-7_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22063-0
Online ISBN: 978-3-031-22064-7
eBook Packages: Computer ScienceComputer Science (R0)