Abstract
Deep neural networks have achieved remarkable success in various research fields, but they face limitations. Firstly, complex models are often required to handle challenging scenarios. Secondly, limited storage and processing power on mobile devices hinder model training and deployment. To address these challenges, we propose a novel approach using a compact and efficient student model to learn from a cumbersome teacher model. To enhance feature map information extraction, we introduce an attention structure that leverages the rich features in the teacher model’s feature maps. Adversarial training is incorporated by treating the student model as a generator and employing a discriminator to differentiate between teacher and student feature maps. Through an iterative process, the student model’s feature map gradually approximates that of the teacher while improving the discriminator’s discrimination abilities. By leveraging the knowledge of the teacher model and incorporating attention mechanisms and adversarial training, our approach provides a compelling solution to the challenges of complex model architectures and limited hardware resources. It achieves impressive performance enhancements with the student model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Goyal, A., Bochkovskiy, A., Deng, J., Koltun, V.: Non-deep networks. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Processing Systems, vol. 35, pp. 6789–6801. Curran Associates Inc. (2022)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv:1503.02531 (2015)
Lopez-Paz, D., Bottou, L., Schölkopf, B., Vapnik, V.: Unifying distillation and privileged information. arXiv preprint arXiv:1511.03643 (2015)
Ba, J., Caruana, R.: Do deep nets really need to be deep? Adv. Neural Inf. Process. Syst. 27 (2014)
Heo, B., Lee, M., Yun, S., Choi, J.Y.: Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 3779–3787 (2019)
Zhou, C., Neubig, G., Gu, J.: Understanding knowledge distillation in non-autoregressive machine translation. arXiv preprint arXiv:1911.02727 (2019)
Guo, M.-H., et al.: Attention mechanisms in computer vision: a survey. Comput. Vis. Media 8(3), 331–368 (2022)
Goodfellow, I., et al.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020)
Zagoruyko, S., Komodakis, N.: Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928 (2016)
Zhong, Z., et al.: Squeeze-and-attention networks for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13065–13074 (2020)
Ulutan, O., Iftekhar, A., Manjunath, B.S.: Vsgnet: spatial attention network for detecting human object interactions using graph convolutions, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13617–13626 (2020)
Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)
Brock, A., Donahue, J., Simonyan, K.: Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018)
Ding, X., Wang, Y., Xu, Z., Wang, Z.J., Welch, W.J.: Distilling and transferring knowledge via cgan-generated samples for image classification and regression. Expert Syst. Appl. 213, 119060 (2023)
Fang, G., Song, J., Shen, C., Wang, X., Chen, D., Song, M.: Data-free adversarial distillation. arXiv preprint arXiv:1912.11006 (2019)
Chen, H., et al.: Data-free learning of student networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3514–3522 (2019)
Chen, H., et al.: Distilling portable generative adversarial networks for image translation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 3585–3592 (2020)
Wang, X., Zhang, R., Sun, Y., Qi, J.: Kdgan: knowledge distillation with generative adversarial networks. Adv. Neural Inf. Process. Syst. 31 (2018)
Chung, I., Park, S., Kim, J., Kwak, N.: Feature-map-level online adversarial knowledge distillation. In: International Conference on Machine Learning, pp. 2006–2015. PMLR (2020)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.P.: An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32
Krizhevsky, A., et al.: Learning multiple layers of features from tiny images (2009)
Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)
Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014)
Kim, J., Park, S., Kwak, N.: Paraphrasing complex network: network compression via factor transfer. Adv. Neural Inf. Process. Syst. 31 (2018)
Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grant No. 61877009, No. 62276054 and the Sichuan Science and Technology Program under contract number 2023YFG0156.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zheng, F., Zuo, L., Guo, F., Luo, W., Hu, Y. (2024). AAKD-Net: Attention-Based Adversarial Knowledge Distillation Network for Image Classification. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1961. Springer, Singapore. https://doi.org/10.1007/978-981-99-8126-7_26
Download citation
DOI: https://doi.org/10.1007/978-981-99-8126-7_26
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8125-0
Online ISBN: 978-981-99-8126-7
eBook Packages: Computer ScienceComputer Science (R0)