skip to main content
10.1145/3485832.3485881acmotherconferencesArticle/Chapter ViewAbstractPublication PagesacsacConference Proceedingsconference-collections
research-article

Understanding the Threats of Trojaned Quantized Neural Network in Model Supply Chains

Published: 06 December 2021 Publication History

Abstract

Deep learning with edge computing arises as a popular paradigm for powering edge devices with intelligence. As the size of deep neural networks (DNN) continually increases, model quantization, which converts the full-precision model into lower-bit representation while mostly preserving the accuracy, becomes a prerequisite for deploying a well-trained DNN on resource-limited edge devices. However, to properly quantize a DNN requires an essential amount of expert knowledge, or otherwise the model accuracy would be devastatingly affected. Alternatively, recent years witness the birth of third-party model supply chains which provide pretrained quantized neural networks (QNN) for free downloading.
In this paper, we systematically analyze the potential threats of trojaned models in third-party QNN supply chains. For the first time, we describe and implement a QUAntization-SpecIfic backdoor attack (QUASI), which manipulates the quantization mechanism to inject a backdoor specific to the quantized model. In other words, the attacker-specified inputs, or triggers, would not cause misbehaviors of the trojaned model in full precision until the backdoor function is automatically completed by a normal quantization operation, producing a trojaned QNN which can be triggered with a near 100% success rate. Our proposed QUASI attack reveals several key vulnerabilities in the existing QNN supply chains: (i) QUASI demonstrates a third-party QNN released online can also be injected with backdoors, while, unlike full-precision models, there is almost no working algorithm for checking the fidelity of a QNN. (ii) More threateningly, the backdoor injected by QUASI remains inactivated in the full-precision model, which inhibits model consumers from attributing undergoing trojan attacks to the malicious model provider. As a practical implication, we alarm it can be highly risky to accept and deploy third-party QNN on edge devices at the current stage, if without future mitigation studies.

References

[1]
[n.d.]. Android Demo Apps - PyTorch. https://github.com/pytorch/android-demo-app. Accessed: 2021-05-21.
[2]
[n.d.]. Available Models-PaddleLite. https://paddle-lite.readthedocs.io/zh/latest/introduction/support_model_list.html. Accessed: 2021-05-21.
[3]
[n.d.]. Models - Machine Learing - Apple Developer. https://developer.apple.com/machine-learning/models/. Accessed: 2021-05-21.
[4]
[n.d.]. Quantization-PyTorch. https://pytorch.org/docs/stable/quantization.html. Accessed: 2021-05-21.
[5]
[n.d.]. Quantization-Tensorflow. https://www.tensorflow.org/model_optimization/guide/quantization/post_training. Accessed: 2021-05-21.
[6]
[n.d.]. US Government’s TrojAI Program. https://www.iarpa.gov/index.php/research-programs/trojai. Accessed: 2021-02-01.
[7]
Martín Abadi, P. Barham, J. Chen, and et al.2016. TensorFlow: A system for large-scale machine learning. In OSDI.
[8]
E. Bagdasaryan and Vitaly Shmatikov. 2021. Blind Backdoors in Deep Learning Models. USENIX Security Symposium(2021).
[9]
E. Bagdasaryan, Andreas Veit, Yiqing Hua, D. Estrin, and Vitaly Shmatikov. 2020. How To Backdoor Federated Learning. In AISTATS.
[10]
Xiaoyu Cao, J. Jia, and N. Gong. 2021. IPGuard: Protecting Intellectual Property of Deep Neural Networks via Fingerprinting the Classification Boundary. Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security (2021).
[11]
Nicholas Carlini and David A. Wagner. 2017. Towards Evaluating the Robustness of Neural Networks. 2017 IEEE Symposium on Security and Privacy (SP) (2017), 39–57.
[12]
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Ben Edwards, Taesung Lee, Ian Molloy, and B. Srivastava. 2019. Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering. ArXiv abs/1811.03728(2019).
[13]
Huili Chen, Cheng Fu, J. Zhao, and F. Koushanfar. 2019. DeepInspect: A Black-box Trojan Detection and Mitigation Framework for Deep Neural Networks. In IJCAI.
[14]
Jiasi Chen and Xukan Ran. 2019. Deep Learning With Edge Computing: A Review. Proc. IEEE 107(2019), 1655–1674.
[15]
X. Chen, Chang Liu, Bo Li, Kimberly Lu, and D. Song. 2017. Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. ArXiv (2017).
[16]
Matthieu Courbariaux, Yoshua Bengio, and J. David. 2015. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. In NIPS.
[17]
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. NeurIPS (2016).
[18]
Bao Gia Doan, Ehsan Abbasnejad, and D. Ranasinghe. 2020. Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems. ACSAC (2020).
[19]
Min Du, R. Jia, and D. Song. 2020. Robust Anomaly Detection and Backdoor Attack Detection Via Differential Privacy. ICLR (2020).
[20]
Kirsty Duncan, E. Komendantskaya, Rob Stewart, and M. Lones. 2020. Relative Robustness of Quantized Neural Networks Against Adversarial Attacks. 2020 International Joint Conference on Neural Networks (IJCNN) (2020), 1–8.
[21]
Yansong Gao, Chang Xu, Derui Wang, S. Chen, D. Ranasinghe, and S. Nepal. 2019. STRIP: a defence against trojan attacks on deep neural networks. ACSAC (2019).
[22]
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, M. Mahoney, and K. Keutzer. 2021. A Survey of Quantization Methods for Efficient Neural Network Inference. ArXiv abs/2103.13630(2021).
[23]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.
[24]
I. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. CoRR abs/1412.6572(2015).
[25]
R. Gray and D. Neuhoff. 1998. Quantization. IEEE Trans. Inf. Theory 44 (1998), 2325–2383.
[26]
Tianyu Gu, K. Liu, Brendan Dolan-Gavitt, and S. Garg. 2019. BadNets: Evaluating Backdooring Attacks on Deep Neural Networks. IEEE Access (2019).
[27]
Wenbo Guo, Lun Wang, Yan Xu, Xinyu Xing, Min Du, and D. Song. 2020. Towards Inspecting and Eliminating Trojan Backdoors in Deep Neural Networks. ICDM (2020).
[28]
Kartik Gupta and Thalaiyasingam Ajanthan. 2020. Improved Gradient based Adversarial Attacks for Quantized Networks. ArXiv abs/2003.13511(2020).
[29]
Sanghyun Hong, Pietro Frigo, Yigitcan Kaya, Cristiano Giuffrida, and T. Dumitras. 2019. Terminal Brain Damage: Exposing the Graceless Degradation in Deep Neural Networks Under Hardware Fault Attacks. In USENIX Security Symposium.
[30]
Benoit Jacob, S. Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and D. Kalenichenko. 2018. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 2704–2713.
[31]
Yujie Ji, Xinyang Zhang, Shouling Ji, X. Luo, and Ting Wang. 2018. Model-Reuse Attacks on Deep Learning Systems. Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (2018).
[32]
Elias Boutros Khalil, Amrita Gupta, and B. Dilkina. 2019. Combinatorial Attacks on Binarized Neural Networks. ICLR (2019).
[33]
Yoongu Kim, Ross Daly, Jeremie S. Kim, Chris Fallin, Ji-Hye Lee, Donghyuk Lee, C. Wilkerson, K. Lai, and O. Mutlu. 2014. Flipping bits in memory without accessing them: An experimental study of DRAM disturbance errors. 2014 ACM/IEEE 41st International Symposium on Computer Architecture (ISCA) (2014), 361–372.
[34]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980(2015).
[35]
Raghuraman Krishnamoorthi. 2018. Quantizing deep convolutional networks for efficient inference: A whitepaper. ArXiv abs/1806.08342(2018).
[36]
A. Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images.
[37]
Yann LeCun, Léon Bottou, Yoshua Bengio, 1998. Gradient-based learning applied to document recognition.
[38]
Fengfu Li and Bin Liu. 2016. Ternary Weight Networks. NeurIPS (2016).
[39]
Shaofeng Li, Minhui Xue, B. Zhao, H. Zhu, and Xinpeng Zhang. 2019. Invisible Backdoor Attacks on Deep Neural Networks via Steganography and Regularization. TDSC (2019).
[40]
Ji Lin, Chuang Gan, and Song Han. 2019. Defensive Quantization: When Efficiency Meets Robustness. ArXiv abs/1904.08444(2019).
[41]
Junyu Lin, Lei Xu, Yingqi Liu, and X. Zhang. 2020. Composite Backdoor Attack for Deep Neural Network by Mixing Existing Benign Features. Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security (2020).
[42]
K. Liu, Brendan Dolan-Gavitt, and S. Garg. 2018. Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks. In RAID.
[43]
Y. Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and X. Zhang. 2019. ABS: Scanning Neural Networks for Back-doors by Artificial Brain Stimulation. CCS (2019).
[44]
Yingqi Liu, Shiqing Ma, Yousra Aafer, W. Lee, Juan Zhai, Weihang Wang, and X. Zhang. 2018. Trojaning Attack on Neural Networks. NDSS (2018).
[45]
L. V. D. Maaten and Geoffrey E. Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9 (2008), 2579–2605.
[46]
Alberto G. Matachana, Kenneth T. Co, Luis Muñoz-González, David Martínez, and Emil C. Lupu. 2020. Robustness and Transferability of Universal Attacks on Compressed Models. ArXiv abs/2012.06024(2020).
[47]
A. Nguyen and A. Tran. 2020. Input-Aware Dynamic Backdoor Attack. NeurIPS (2020).
[48]
Adam Paszke, S. Gross, Francisco Massa, and et al.2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS.
[49]
Ximing Qiao, Yukun Yang, and Hongbing Li. 2019. Defending Neural Backdoors via Generative Distribution Modeling. In NeurIPS.
[50]
A. S. Rakin, Zhezhi He, and Deliang Fan. 2019. Bit-Flip Attack: Crushing Neural Network With Progressive Bit Search. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019), 1211–1220.
[51]
A. S. Rakin, Zhezhi He, and Deliang Fan. 2020. TBT: Targeted Neural Network Attack With Bit Trojan. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), 13195–13204.
[52]
Kaveh Razavi, Ben Gras, E. Bosman, B. Preneel, Cristiano Giuffrida, and H. Bos. 2016. Flip Feng Shui: Hammering a Needle in the Software Stack. In USENIX Security Symposium.
[53]
Cyril Roscian, A. Sarafianos, J. Dutertre, and A. Tria. 2013. Fault Model Analysis of Laser-Induced Faults in SRAM Memory Cells. 2013 Workshop on Fault Diagnosis and Tolerance in Cryptography (2013), 89–98.
[54]
A. Salem, Rui Wen, M. Backes, Shiqing Ma, and Y. Zhang. 2020. Dynamic Backdoor Attacks Against Machine Learning Models. ArXiv (2020).
[55]
K. Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556(2015).
[56]
J. Stallkamp, Marc Schlipsing, J. Salmen, and C. Igel. 2012. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks : the official journal of the International Neural Network Society 32(2012), 323–32.
[57]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, D. Erhan, I. Goodfellow, and R. Fergus. 2014. Intriguing properties of neural networks. CoRR abs/1312.6199(2014).
[58]
Brandon Tran, Jerry Li, and A. Madry. 2018. Spectral Signatures in Backdoor Attacks. In NeurIPS.
[59]
Alexander Turner, D. Tsipras, and A. Madry. 2019. Label-Consistent Backdoor Attacks. ArXiv (2019).
[60]
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, B. Viswanath, H. Zheng, and B. Zhao. 2019. Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks. Security & Privacy (2019).
[61]
Jiaxiang Wu, C. Leng, Yuhang Wang, Q. Hu, and Jian Cheng. 2016. Quantized Convolutional Neural Networks for Mobile Devices. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 4820–4828.
[62]
Yun Xiang, Zhuang-Zhi Chen, Zuohui Chen, Zebin Fang, Haiyang Hao, Jinyin Chen, Yi Liu, Zhefu Wu, Qi Xuan, and Xiaoniu Yang. 2020. Open DNN Box by Power Side-Channel Attack. IEEE Transactions on Circuits and Systems II: Express Briefs 67 (2020), 2717–2721.
[63]
Mengjia Yan, Christopher W. Fletcher, and J. Torrellas. 2020. Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN Architectures. USENIX Security (2020).
[64]
Jiancheng Yang, R. Shi, and Bingbing Ni. 2021. MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis. 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI) (2021), 191–195.
[65]
Fan Yao, A. S. Rakin, and Deliang Fan. 2020. DeepHammer: Depleting the Intelligence of Deep Neural Networks through Targeted Chain of Bit Flips. In USENIX Security Symposium.
[66]
Yuanshun Yao, Huiying Li, H. Zheng, and B. Zhao. 2019. Latent Backdoor Attacks on Deep Neural Networks. CCS (2019).
[67]
Honggang Yu, Haocheng Ma, Kaichen Yang, Yiqiang Zhao, and Yier Jin. 2020. DeepEM: Deep Neural Networks Model Recovery through EM Side-Channel Information Leakage. 2020 IEEE International Symposium on Hardware Oriented Security and Trust (HOST) (2020), 209–218.

Cited By

View all
  • (2024)Purifying quantization-conditioned backdoors via layer-wise activation correction with distribution approximationProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693165(27439-27456)Online publication date: 21-Jul-2024
  • (2024)Backdoor Learning: A SurveyIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.318297935:1(5-22)Online publication date: Jan-2024
  • (2024)BELT: Old-School Backdoor Attacks can Evade the State-of-the-Art Defense with Backdoor Exclusivity Lifting2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00226(2124-2141)Online publication date: 19-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ACSAC '21: Proceedings of the 37th Annual Computer Security Applications Conference
December 2021
1077 pages
ISBN:9781450385794
DOI:10.1145/3485832
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 December 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. backdoor attack
  2. model quantization
  3. neural network

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ACSAC '21

Acceptance Rates

Overall Acceptance Rate 104 of 497 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)72
  • Downloads (Last 6 weeks)2
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Purifying quantization-conditioned backdoors via layer-wise activation correction with distribution approximationProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693165(27439-27456)Online publication date: 21-Jul-2024
  • (2024)Backdoor Learning: A SurveyIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.318297935:1(5-22)Online publication date: Jan-2024
  • (2024)BELT: Old-School Backdoor Attacks can Evade the State-of-the-Art Defense with Backdoor Exclusivity Lifting2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00226(2124-2141)Online publication date: 19-May-2024
  • (2024)Towards robustness evaluation of backdoor defense on quantized deep learning modelsExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.124599255:PBOnline publication date: 18-Oct-2024
  • (2024)Data Poisoning Quantization Backdoor AttackComputer Vision – ECCV 202410.1007/978-3-031-72907-2_3(38-54)Online publication date: 31-Oct-2024
  • (2024)Certified Quantization Strategy Synthesis for Neural NetworksFormal Methods10.1007/978-3-031-71162-6_18(343-362)Online publication date: 9-Sep-2024
  • (2023)Deep Neural Network Quantization Framework for Effective Defense against Membership Inference AttacksSensors10.3390/s2318772223:18(7722)Online publication date: 7-Sep-2023
  • (2023)Quantization Backdoors to Deep Learning Commercial FrameworksIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2023.327195621:3(1155-1172)Online publication date: 1-May-2023
  • (2023)Ten years after ImageNet: a 360° perspective on artificial intelligenceRoyal Society Open Science10.1098/rsos.22141410:3Online publication date: 29-Mar-2023
  • (2022)Towards Model Quantization on the Resilience Against Membership Inference Attacks2022 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP46576.2022.9897681(3646-3650)Online publication date: 16-Oct-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media