Abstract
This paper focuses on the problem of transplanting category-and-task-specific neural networks to a generic, modular network without strong supervision. Unlike traditional deep neural networks (DNNs) with black-box representations, we design a functionally modular network architecture, which divides the entire DNN into several functionally meaningful modules. Like building LEGO blocks, we can teach the proposed DNN a new object category by directly transplanting the module corresponding to the object category from another DNN, with a few or even without sample annotations. Our method incrementally adds new categories to the DNN, which do not affect representations of existing categories. Such a strategy of incremental network transplanting can avoid the typical catastrophic-forgetting problem in continual learning. We further develop a back distillation method to overcome challenges of model optimization in network transplanting. In experiments, our method with much fewer training samples outperformed baselines.
Q. Zhang and X. Cheng—Contributed equally to this paper.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aljundi, R., Lin, M., Goujaud, B., Bengio, Y.: Gradient based sample selection for online continual learning. In NIPS
Andrychowicz, M., Denil, M., Colmenarejo, S.G., Hoffman, M.W., Pfau, D., Schaul, T., Shillingford, B., de Freitas, N.: Learning to learn by gradient descent by gradient descent. In NIPS (2016)
Bau, D., Zhou, B., Khosla, A., Oliva, A., Torralba, A.: Network dissection: Quantifying interpretability of deep visual representations. In CVPR (2017)
Buzzega, P., Boschini, M., Porrello, A., Abati, D., Calderara, S.: Dark experience for general continual learning: a strong, simple baseline. In NIPS
Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., Su, J.K.: This looks like that: deep learning for interpretable image recognition. In NIPS (2019)
Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., Yuille, A.: Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR (2014)
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS (2016)
Chou, Y.M., Chan, Y.M., Lee, J.H., Chiu, C.Y., Chen, C.S.: Unifying and merging well-trained deep neural networks for inference stage. In arXiv:1805.04980 (2018)
Hariharan, B., Arbelaez, P., Bourdev, L., Maji, S., Malik, J.: Semantic contours from inverse detectors. In ICCV (2011)
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., Lerchner, A.: \(\beta \)-vae: learning basic visual concepts with a constrained variational framework. In ICLR (2017)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In NIPS Workshop (2014)
Hospedales, T., Antoniou, A., Micaelli, P., Storkey, A.: Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence 44(9), 5149–5169 (2021)
Long, J., Shelhamer, E., Darrel, T.: Fully convolutional networks for semantic segmentation. In CVPR (2015)
Metz, L., Maheswaranathan, N., Cheung, B., Sohl-Dickstein, J.: Learning unsupervised learning rules. In ICLR (2019)
Neyshabur, B., Sedghi, H., Zhang, C.: What is being transferred in transfer learning? In NIPS
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR
Rebuffi, S.A., Bilen, H., Vedaldi, A.: Learning multiple visual domains with residual adapters. In NIPS (2017)
Rebuffi, S.A., Bilen, H., Vedaldi, A.: Efficient parametrization of multi-domain deep neural networks. In CVPR (2018)
Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. In arXiv:1606.04671 (2016)
Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In NIPS (2017)
Schwartz-Ziv, R., Tishby, N.: Opening the black box of deep neural networks via information. In arXiv:1703.00810 (2017)
Shen, W., Wei, Z., Huang, S., Zhang, B., Fan, J., Zhao, P., Zhang, Q.: Interpretable compositional convolutional neural networks. In IJCAI (2021)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In ICLR (2015)
Srinivas, S., Fleuret, F.: Knowledge transfer with jacobian matching. In ICML (2018)
Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset. Tech. rep, In California Institute of Technology (2011)
Wang, Y., Su, H., Zhang, B., Hu, X.: Interpret neural networks by identifying critical data routing paths. In CVPR (2018)
Wolchover, N.: New theory cracks open the black box of deep learning. In Quanta Magazine (2017)
Xiao, J., Gu, S., Zhang, L.: Multi-domain learning for accurate and few-shot color constancy (2020)
Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. In ICLR (2018)
Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In NIPS (2014)
Zhang, Q., Wu, Y.N., Zhu, S.C.: Interpretable convolutional neural networks. In CVPR (2018)
Acknowledgment
Quanshi Zhang and Xu Cheng contribute equally to this paper. Quanshi Zhang is the corresponding author. He is with the Department of Computer Science and Engineering, the John Hopcroft Center, at the Shanghai Jiao Tong University, China. This work is partially supported by the National Nature Science Foundation of China (62276165), National Key R &D Program of China (2021ZD0111602), Shanghai Natural Science Foundation (21JC1403800,21ZR1434600), National Nature Science Foundation of China (U19B2043).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhang, Q., Cheng, X., Wang, X., Yang, Y., Wu, Y. (2024). Network Transplanting for the Functionally Modular Architecture. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14427. Springer, Singapore. https://doi.org/10.1007/978-981-99-8435-0_6
Download citation
DOI: https://doi.org/10.1007/978-981-99-8435-0_6
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8434-3
Online ISBN: 978-981-99-8435-0
eBook Packages: Computer ScienceComputer Science (R0)