Abstract
Practical applications of deep learning are challenged with critical issues that the distributions of training data and testing data are different and the labels of testing data are insufficient. To address these problems, unsupervised domain adaptation (UDA) based transfer learning has gained significant attention. However, advanced deep learning models of UDA are too complex for real-time and resource-constrained applications. In this paper, we propose an iterative transfer model compression (ITMC) method with two key modules, i.e., transfer knowledge distillation (TKD) and adaptive channel pruning (ACP). During each epoch, the TKD module achieves model compression by distilling the knowledge from the teacher model to the student model, while facilitating the transfer of cross-domain knowledge to enhance the performance in the target domain. Concurrently, with the aid of the ACP module, redundant channels in the student model are pruned to reduce the computational cost while retaining the model accuracy. In particular, the alternation of ACP and TKD ensures effective knowledge transfer, balancing the model size and its performance in the target domain. Experimental results demonstrate that ITMC approach achieves higher accuracy under the same compression ratio compared with the state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhang, Z., et al.: Object relational graph with teacher-recommended learning for video captioning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13275–13285 (2020). https://doi.org/10.1109/CVPR42600.2020.01329
Zhuang, F., et al.: A comprehensive survey on transfer learning. Proc. IEEE 109(1), 43–76 (2020)
Du, J., Nie, T., Dou, W., Shen, D., Kou, Y.: SAREM: semi-supervised active heterogeneous entity matching framework. In: Zhao, X., Yang, S., Wang, X., Li, J. (eds.) WISA 2022. LNCS, vol. 13579, pp. 77–88. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20309-1_7
Dai, W., Yang, Q., Xue, G.R., Yu, Y.: Boosting for transfer learning. In: Proceedings of the 24th International Conference on Machine Learning, pp. 193–200 (2007)
Borgwardt, K.M., Gretton, A., Rasch, M.J., Kriegel, H.P., Schölkopf, B., Smola, A.J.: Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22(14), 49–57 (2006)
Mei, Z., Cai, K., Shi, L., Li, J., Chen, L., Immink, K.A.S.: Deep transfer learning-based detection for flash memory channels. IEEE Trans. Commun. (2024)
Ganin, Y., et al.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17(1), 1–35 (2016)
Yang, J., et al.: Quantization networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7308–7316 (2019)
Molchanov, P., Mallya, A., Tyree, S., Frosio, I., Kautz, J.: Importance estimation for neural network pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264–11272 (2019)
Blalock, D., Gonzalez Ortiz, J.J., Frankle, J., Guttag, J.: What is the state of neural network pruning? Proc. Mach. Learn. Syst. 2, 129–146 (2020)
Idelbayev, Y., Carreira-Perpinán, M.A.: Low-rank compression of neural nets: learning the rank of each layer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8049–8059 (2020)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: a survey. Int. J. Comput. Vision 129, 1789–1819 (2021)
Prakosa, S.W., Leu, J.S., Chen, Z.H.: Improving the accuracy of pruned network using knowledge distillation. Pattern Anal. Appl. 24, 819–830 (2021)
Yu, C., Wang, J., Chen, Y., Wu, Z.: Accelerating deep unsupervised domain adaptation with transfer channel pruning. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2019)
Ruder, S., Ghaffari, P., Breslin, J.G.: Knowledge adaptation: teaching to adapt. arXiv preprint arXiv:1702.02052 (2017)
Granger, E., Kiran, M., Dolz, J., Blais-Morin, L.A., et al.: Joint progressive knowledge distillation and unsupervised domain adaptation. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2020)
Ho, Y., Wookey, S.: The real-world-weight cross-entropy loss function: modeling the costs of mislabeling. IEEE Access 8, 4806–4813 (2019)
Tammina, S.: Transfer learning using VGG-16 with deep convolutional neural network for classifying images. Int. J. Sci. Res. Publ. (IJSRP) 9(10), 143–150 (2019)
Targ, S., Almeida, D., Lyman, K.: Resnet in resnet: generalizing residual architectures. arXiv preprint arXiv:1603.08029 (2016)
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China (No. 62371239, No. 62201258, No. 62272469, No. 62202232), in part by the Natural Science Foundation of Jiangsu Province under Grant BK20210331, in part by the Jiangsu Specially-Appointed Professor Program 2021, in part by the open research fund of National Mobile Communications Research Laboratory, Southeast University (No. 2023D12), in part by the Fundamental Research Funds for the Central Universities (No. 30923011035), in part by the Science and Technology Innovation Program of Hunan Province (No. 2023RC1007).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, Z., Shi, L., Mei, Z., Zhao, X., Wang, Z., Li, J. (2024). Iterative Transfer Knowledge Distillation and Channel Pruning for Unsupervised Cross-Domain Compression. In: Jin, C., Yang, S., Shang, X., Wang, H., Zhang, Y. (eds) Web Information Systems and Applications. WISA 2024. Lecture Notes in Computer Science, vol 14883. Springer, Singapore. https://doi.org/10.1007/978-981-97-7707-5_1
Download citation
DOI: https://doi.org/10.1007/978-981-97-7707-5_1
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-7706-8
Online ISBN: 978-981-97-7707-5
eBook Packages: Computer ScienceComputer Science (R0)