Abstract
This paper mainly studies on the potential safety hazards in the obstacle recognition and processing system (ORPS) of the self-driving cars, which is constructed by deep learning architecture. We perform an attack that embeds a backdoor in the Mask R-CNN in ORPS by poisoning the dataset. Under normal circumstances, the backdoored model can accurately identify obstacles (vehicles). However, under certain circumstances, triggering the backdoor in the backdoored model may lead to change the size (bounding box and mask) and confidence of the detected obstacles, which may cause serious accidents. The experiment result shows that it is possible to embed a backdoor in ORPS. We can see that the backdoored network can obviously change the size of bounding box and corresponding mask of those poisoned instances. But on the other hand, embedding a backdoor in the deep learning based model will only slightly affect the accuracy of detecting objects without backdoor triggers, which is imperceptible for users. Eventually, we hope that our simple work can arouse people’s attention to the self-driving technology and even other deep learning based models. It brings motivation about how to judge or detect the existence of the backdoors in these systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, X., Liu, C., Li, B., Lu, K., Song, D.: Targeted backdoor attacks on deep learning systems using data poisoning (2017)
Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding (2016)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2009. pp. 248–255. IEEE (2009)
Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The PASCAL visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. ArXiv e-prints, December 2014
Gu, T., Dolan-Gavitt, B., Garg, S.: BadNets: identifying vulnerabilities in the machine learning model supply chain. CoRR abs/1708.06733 (2017). http://arxiv.org/abs/1708.06733
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. ArXiv e-prints, March 2017
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Koh, P.W., Liang, P.: Understanding black-box predictions via influence functions (2017)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
LeCun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)
Liu, Y., et al.: Trojaning attack on neural networks. In: Network and Distributed System Security Symposium (2017)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Moosavidezfooli, S.M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations, pp. 86–94 (2016)
Muñoz-González, L., et al.: Towards poisoning of deep learning algorithms with back-gradient optimization. ArXiv e-prints, August 2017
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)
Papernot, N., Mcdaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning, pp. 506–519 (2016)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: OverFeat: integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229 (2013)
Shen, S., Tople, S., Saxena, P.: A uror: defending against poisoning attacks in collaborative deep learning systems. In: Proceedings of the 32nd Annual Conference on Computer Security Applications, pp. 508–519. ACM (2016)
Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)
Timofte, R., Zimmermann, K., Gool, L.V.: Multi-view traffic sign detection, recognition, and 3D localisation. Mach. Vis. Appl. 25(3), 633–647 (2014)
Yang, C., Wu, Q., Li, H., Chen, Y.: Generative poisoning attack method against neural networks (2017)
Yang, F., Choi, W., Lin, Y.: Exploit all the layers: fast and accurate CNN object detector with scale dependent pooling and cascaded rejection classifiers. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2129–2137 (2016)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53
Zou, M., Shi, Y., Wang, C., Li, F., Song, W.Z., Wang, Y.: PoTrojan: powerful neural-level trojan designs in deep learning models (2018)
Acknowledgement
This paper is partially supported by the National Natural Science Foundation of China grants 61772147, and the Key Basic Research of Guangdong Province Natural Science Fund Fostering Projects grants 2015A030308016 and National Climb – B Plan (Grand No. pdjhb0400).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Wu, J., Lin, X., Lin, Z., Tang, Y. (2018). A Security Concern About Deep Learning Models. In: Liu, F., Xu, S., Yung, M. (eds) Science of Cyber Security. SciSec 2018. Lecture Notes in Computer Science(), vol 11287. Springer, Cham. https://doi.org/10.1007/978-3-030-03026-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-03026-1_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03025-4
Online ISBN: 978-3-030-03026-1
eBook Packages: Computer ScienceComputer Science (R0)