Abstract
Transfer Learning and Fine-Tuning are learning frameworks for dealing with the lack of labeled data in Deep Learning. These methods work effectively in a Convolutional Neural Network (CNN) that acquires common features in the convolutional layers. When transferring CNN layers, it is common to transfer multiple convolutional layers close to the input side excluding the identification layer after training the source task. On the other hand, there are few studies that focus on transfer during learning and higher-level transfer. In this paper, we propose Fine-Tuning by transferring convolutional filters during learning. Filters are ranked using pruning criteria, and only the low importance filters are overwritten as target filters. In the 10-class classification using a subset of CIFAR-100, we show that the proposed method can improve the test accuracy by up to 2% compared to training from scratch. We also show the tendency of the ratio of low importance filters for each layer changes with the progress of learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: ECCV (2014)
Krizhevsky, A., Nair, V., Hinto, G.: The CIFAR-10 dataset. https://www.cs.toronto.edu/kriz/cifar.html. Accessed 12 July 2019
Zhu, X.: Semi-supervised learning literature survey. Technical report 1530, Computer Sciences, University of Wisconsin-Madison (2005)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE CVPR (2009)
Rosenstein, M.T., Marx, Z., Kaelbling, L.P., Dietterich, T.G.: To transfer or not to transfer. In: NIPS 2005 Workshop on Transfer Learning (2005)
Zamir, A., Sax, A., Shen, W., Guibas, L., Malik, J., Savarese, S.: Taskonomy: disentangling task transfer learning. In: IEEE CVPR (2018)
Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A.A., Pritzel, A., Wierstra, D.: Pathnet: evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 (2017)
Caruana, R.: Multitask learning: a knowledge-based source of inductive bias. In: Proceedings of the Tenth International Conference on machine Learning, University of Massachusetts (1993)
Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. arXiv preprint arXiv:1602.01783 (2016)
Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. In: ICLR (2017)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)
Xavier, G., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: AISTATS (2011)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML (2015)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. In: JMLR (2014)
Acknowledgements
The authors would like to thank Dr. Ryuji Mine, Mr. Tadayuki Matsumura and Dr. Atsushi Miyamoto from Hitachi Ltd., for their feedback. This research is partly supported by the collaborative research program 2018, Hitachi Kyoto University Laboratory, Center for Exploratory Research, Hitachi Ltd. Advices given by M. Sato from Tokai University has been also a great help in writing the paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ikawa, S., Sato, Y. (2020). Multi-task Learning Using Online Fine-Tuning Considering the Importance of Each Filter. In: Sato, H., Iwanaga, S., Ishii, A. (eds) Proceedings of the 23rd Asia Pacific Symposium on Intelligent and Evolutionary Systems. IES 2019. Proceedings in Adaptation, Learning and Optimization, vol 12. Springer, Cham. https://doi.org/10.1007/978-3-030-37442-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-37442-6_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37441-9
Online ISBN: 978-3-030-37442-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)