research-article

Free access

DeepXplore: automated whitebox testing of deep learning systems

Authors:

Suman JanaAuthors Info & Claims

Communications of the ACM, Volume 62, Issue 11

Pages 137 - 145

https://doi.org/10.1145/3361566

Published: 24 October 2019 Publication History

All formats PDF

Abstract

Deep learning (DL) systems are increasingly deployed in safety- and security-critical domains such as self-driving cars and malware detection, where the correctness and predictability of a system's behavior for corner case inputs are of great importance. Existing DL testing depends heavily on manually labeled data and therefore often fails to expose erroneous behaviors for rare inputs.

We design, implement, and evaluate DeepXplore, the first white-box framework for systematically testing real-world DL systems. First, we introduce neuron coverage for measuring the parts of a DL system exercised by test inputs. Next, we leverage multiple DL systems with similar functionality as cross-referencing oracles to avoid manual checking. Finally, we demonstrate how finding inputs for DL systems that both trigger many differential behaviors and achieve high neuron coverage can be represented as a joint optimization problem and solved efficiently using gradient-based search techniques.

DeepXplore efficiently finds thousands of incorrect corner case behaviors (e.g., self-driving cars crashing into guard rails and malware masquerading as benign software) in state-of-the-art DL models with thousands of neurons trained on five popular datasets such as ImageNet and Udacity self-driving challenge data. For all tested DL models, on average, DeepXplore generated one test input demonstrating incorrect behavior within one second while running only on a commodity laptop. We further show that the test inputs generated by DeepXplore can also be used to retrain the corresponding DL model to improve the model's accuracy by up to 3%.

References

[1]

Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604,07316 (2016).

[2]

Brubaker, C., Jana, S., Ray, B., Khurshid, S., Shmatikov V. Using frankencerts for automated adversarial testing of certificate validation in SSL/TLS implementations. In Proceedings of the 35th IEEE Symposium on Security and Privacy (2014).

Digital Library

[3]

Goodfellow, I., Shlens, J., Szegedy, C. Explaining and harnessing adversarial examples. In Proceedings of the 3rd International Conference on Learning Representations (2015).

[4]

Grosse, K., Papernot, N., Manoharan, P., Backes, M., McDaniel, P. Adversarial examples for malware detection. In European Symposium on Research in Computer Security (2017).

[5]

He, K., Zhang, X., Ren, S., Sun, J. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2016).

[6]

Julian, K.D., Lopez, J., Brush, J.S., Owen, M.P., Kochenderfer, M.J. Policy compression for aircraft collision avoidance systems. In Proceedings of the 35th IEEE/AIAA Digital Avionics Systems Conference (2016).

[7]

Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J. Reluplex: An efficient smt solver for verifying deep neural networks. In Proceedings of the 29th International Conference on Computer Aided Verification (2017).

[8]

LeCun, Y., Cortes, C., Burges, C.J. MNIST handwritten digit database. 2010.

[9]

Liu, M.-Y., Breuel, T., Kautz, J. Unsupervised image-to-image translation networks. In Advances in Neural Information Processing Systems (2017).

Digital Library

[10]

Odena, A., Goodfellow, I. Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv:1807.10875 (2018).

[11]

Pei, K., Cao, Y., Yang, J., Jana, S. Towards practical verification of machine learning: The case of computer vision systems. arXiv preprint arXiv:1712.01785 (2017).

[12]

Simonyan, K., Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Proceedings of the 3rd International Conference on Learning Representations (2015).

[13]

Tian, Y., Pei, K., Jana, S., Ray, B. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering, ACM (2018), 303--314

Digital Library

[14]

Šrndic, N., Laskov, P. Practical evasion of a learning-based classifier: a case study. In Proceedings of the 35th IEEE Symposium on Security and Privacy (2014).

[15]

Wang, S., Pei, K., Whitehouse, J., Yang, J., Jana, S. Efficient formal safety analysis of neural networks. In Advances in Neural Information Processing Systems (2018).

[16]

Wang, S., Pei, K., Whitehouse, J., Yang, J., Jana, S. Formal security analysis of neural networks using symbolic intervals. In 27th USENIX Security Symposium (2018).

Digital Library

[17]

Wong, E., Kolter, Z. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning (2018).

[18]

Yosinski, J., Clune, J., Fuchs, T., Lipson, H. Understanding neural networks through deep visualization. In 2015 ICML Workshop on Deep Learning (2015).

Cited By

Rehman FIzurieta C(2025)Testing convolutional neural network based deep learning systems: a statistical metamorphic approachPeerJ Computer Science10.7717/peerj-cs.265811(e2658)Online publication date: 30-Jan-2025
https://doi.org/10.7717/peerj-cs.2658
Wang JPham HLi QTan LGuo YAziz AMeijer E(2025)D³: Differential Testing of Distributed Deep Learning With Model GenerationIEEE Transactions on Software Engineering10.1109/TSE.2024.346165751:1(38-52)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TSE.2024.3461657
Hashemifar SParsa SKalaee A(2025)Path Analysis for Effective Fault Localization in Deep Neural NetworksApplied Soft Computing10.1016/j.asoc.2025.112805(112805)Online publication date: Jan-2025
https://doi.org/10.1016/j.asoc.2025.112805
Show More Cited By

Index Terms

DeepXplore: automated whitebox testing of deep learning systems

Recommendations

DeepXplore: Automated Whitebox Testing of Deep Learning Systems
SOSP '17: Proceedings of the 26th Symposium on Operating Systems Principles

Deep learning (DL) systems are increasingly deployed in safety- and security-critical domains including self-driving cars and malware detection, where the correctness and predictability of a system's behavior for corner case inputs are of great ...
DeepXplore: Automated Whitebox Testing of Deep Learning Systems

Over the past few years, Deep Learning (DL) has made tremendous progress, achieving or surpassing human-level performance for a diverse set of tasks, including image classification, speech recognition, and playing games like Go. These advances have led ...
Deep-Learning Approach with DeepXplore for Software Defect Severity Level Prediction
Computational Science and Its Applications – ICCSA 2021
Abstract
Fixing the defects of earlier releases and working on fast and efficient fixing of those software defects is detrimental for the release of further versions. Bug tracking systems like Bugzilla get thousands of software defect reports every day. ...

Comments

Information & Contributors

Information

Published In

cover image Communications of the ACM

Communications of the ACM Volume 62, Issue 11

November 2019

136 pages

ISSN:0001-0782

EISSN:1557-7317

DOI:10.1145/3368886

Editor:
Andrew A. Chien
Association for Computing Machinery, New York, NY

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2019

Published in CACM Volume 62, Issue 11

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

103
Total Citations
View Citations
12,372
Total Downloads

Downloads (Last 12 months)663
Downloads (Last 6 weeks)119

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rehman FIzurieta C(2025)Testing convolutional neural network based deep learning systems: a statistical metamorphic approachPeerJ Computer Science10.7717/peerj-cs.265811(e2658)Online publication date: 30-Jan-2025
https://doi.org/10.7717/peerj-cs.2658
Wang JPham HLi QTan LGuo YAziz AMeijer E(2025)D³: Differential Testing of Distributed Deep Learning With Model GenerationIEEE Transactions on Software Engineering10.1109/TSE.2024.346165751:1(38-52)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TSE.2024.3461657
Hashemifar SParsa SKalaee A(2025)Path Analysis for Effective Fault Localization in Deep Neural NetworksApplied Soft Computing10.1016/j.asoc.2025.112805(112805)Online publication date: Jan-2025
https://doi.org/10.1016/j.asoc.2025.112805
Xia CHuang SZheng CYang ZBai TSun L(2024)TraModeAVTest: Modeling Scenario and Violation Testing for Autonomous Driving Systems Based on Traffic RegulationsElectronics10.3390/electronics1307119713:7(1197)Online publication date: 25-Mar-2024
https://doi.org/10.3390/electronics13071197
Giunchiglia EImrie Fvan der Schaar MLukasiewicz T(2024)Machine learning with requirements: A manifestoNeurosymbolic Artificial Intelligence10.3233/NAI-240767(1-13)Online publication date: 27-Aug-2024
https://doi.org/10.3233/NAI-240767
Rovira Martorell JGálvez ATirado F(2024)Artificial Intelligence and the Production of Judicial TruthTheory, Culture & Society10.1177/02632764241268174Online publication date: 13-Sep-2024
https://doi.org/10.1177/02632764241268174
Amini MNejati SFilkov VRay BZhou M(2024)Bridging the Gap between Real-world and Synthetic Images for Testing Autonomous Driving SystemsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695067(732-744)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695067
Tambon FKhomh FAntoniol G(2024)GIST: Generated Inputs Sets Transferability in Deep LearningACM Transactions on Software Engineering and Methodology10.1145/367245733:8(1-38)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3672457
Feng XHan XChen SYang W(2024)LLMEffiChecker: Understanding and Testing Efficiency Degradation of Large Language ModelsACM Transactions on Software Engineering and Methodology10.1145/366481233:7(1-38)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3664812
Chen JJia CYan YGe JZheng HCheng Y(2024)A Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning OperatorsProceedings of the ACM on Software Engineering10.1145/36607961:FSE(2005-2027)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660796
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Digital Edition

View this article in digital edition.

Digital Edition

Magazine Site

View this article on the magazine site (external)

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents