Abstract
Numerous applications need to concurrently solve multiple tasks. We present an unsupervised method enabling to create from two pre-trained neural networks A and B, a network B’ approximating B while feeding on a part of A’s layers. This “Vampire” Network allows to significantly reduce the combined weight of the two networks. We propose the following contributions: (1) we show that two networks of the same structure but trained on different tasks display quite strong linear properties between their layers; (2) an unsupervised algorithm replacing part of the vampire network’s features by linear projections of features from the first network; (3) we show that the vampire network thereby created significantly reduces the number of additional parameters needed to accomplish the second task, and thus the computational load of the full system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alhashim, I., Wonka, P.: High quality monocular depth estimation via transfer learning. ArXiv (2019)
Caruana, R.: Multitask learning. Mach. Learn. 28, 41–75 (1997). https://doi.org/10.1023/A:1007379606734
Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)
Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: Advances in Neural Information Processing Systems (2014)
Kozma, R., Ilin, R., Siegelmann, H.T.: Evolution of abstraction across layers in deep learning neural networks. Procedia Comput. Sci. 144, 203–213 (2018)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)
Raghu, M., et al.: SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In: NIPS 2017: Proceedings of the 31st International Conference on Neural Information Processing Systems (2017)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Ruder, S.: An overview of multi-task learning in deep neural networks. ArXiv (2017)
Martin, S., et al.: OnionNet: sharing features in cascaded deep classifiers. In: Proceedings of the British Machine Vision Conference (BMVC) (2016)
Huang, X., et al.: The ApolloScape open dataset for autonomous driving and its application. IEEE Trans. Pattern Anal. Mach. Intell. 42(10), 2702–2719 (2019)
Chou, Y.-M., et al.: Unifying and merging well-trained deep neural networks for inference stage. In: IJCAI: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (2018)
Li, Y., et al.: Convergent learning: do different neural networks learn the same representations? In: ICLR (2016)
Zhang, Y., et al.: An overview of multi-task learning. Natl. Sci. Rev. 5(1), 30–43 (2018)
Zamir, A.R., et al.: Taskonomy: disentangling task transfer learning. In: CVPR: The IEEE Conference on Computer Vision and Pattern Recognition (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Nguyen, TL.R., Chateau, T., Magniez, G. (2022). VampNet: Unsupervised Vampirizing of Convolutional Networks. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2022. Lecture Notes in Computer Science, vol 13598. Springer, Cham. https://doi.org/10.1007/978-3-031-20713-6_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-20713-6_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20712-9
Online ISBN: 978-3-031-20713-6
eBook Packages: Computer ScienceComputer Science (R0)