Abstract
Deep Multi-Task Learning has achieved great success in a number of domains. However, the enormous number of parameters results in extremely large storage costs for current deep Multi-Task models. Several methods based on tensor networks were proposed to address this problem. However, the tensor train format based methods only share the information of one mode. The huge central core tensor of the tucker format is hard to be stored and optimized. To tackle these problems, we introduce a novel Concatenated Tensor Network structure, in particular, Projected Entangled Pair States (PEPS) like, into multi-task models. We name the resulted multi-task models as Concatenated Tensor Multi-Task Learning (CT-MTL).
This work was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100204).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, C., Batselier, K., Ko, C.Y., Wong, N.: Matrix product operator restricted Boltzmann machines. In: 2019 International Joint Conference on Neural Networks(IJCNN), Budapest, pp. 1–8. IEEE (2019)
Chen, X., Hou, M., Zhou, G., Zhao, Q.: Tensor ring nets adapted deep multi-tasklearning (2018). https://openreview.net/forum?id=BJxmXhRcK7
Cichocki, A., Lee, N., Oseledets, I.V., Phan, A.H., Zhao, Q., Mandic, D.P.: Tensornetworks for dimensionality reduction and large-scale optimization: part 1 low-rank tensor decompositions. Found. Trends Mach. Learn. 9(4–5), 249–429 (2016)
Duong, L., Cohn, T., Bird, S., Cook, P.: Low resource dependency parsing: cross-lingual parameter sharing in a neural network parser. In: ACL 2015, Short Papers, pp. 845–850, Beijing (2015)
Huang, H., Liu, Y., Zhu, C.: Low-rank tensor grid for image completion. https://arxiv.org/pdf/1903.04735v3.pdf arXivpreprint arXiv:1903.04735 (2019)
Hübener, R., Nebendahl, V., Dür, W.: Concatenated tensor network states. New J. Phys. 12(2), 025004 (2010)
Long, M., Cao, Z., Wang, J., Philip, S.Y.: Learning multiple tasks with multilinear relationship networks. In: Advances in Neural Information Processing Systems, pp. 1594–1603, Long Beach (2017)
Novikov, A., Podoprikhin, D., Osokin, A., Vetrov, D.P.: Tensorizing neural networks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems, pp. 442–450, Montreal (2015)
Oseledets, I.V.: Tensor-train decomposition. SIAM J. Sci. Comput. 33(5), 2295–2317 (2011)
Pan, Y., et al.: Compressing recurrent neural networks with tensor ring for action recognition. In: AAAI, pp. 4683–4690. AAAI Press (2019)
Verstraete, F., Wolf, M.M., Perez-Garcia, D., Cirac, J.I.: Criticality, the area law, and the computational power of projected entangled pair states. Phys. Rev. Lett. 96(22), 220601 (2006)
Wang, M., Zhang, C., Pan, Y., Xu, J., Xu, Z.: Tensor ring restricted Boltzmann machines. In: IJCNN 2019, pp. 1–8. IEEE (2019)
Yang, Y., Hospedales, T.M.: Deep multi-task representation learning: a tensor factorisation approach. In: ICLR 2017
Yang, Y., Hospedales, T.M.: Trace norm regularised deep multi-task learning. In: Proceedings of ICLR 2017, Workshop Track (2017)
Ye, J., et al.: Learning compact recurrent neural networks with block-term tensor decomposition. In: CVPR 2018, Salt Lake City, pp. 9378–9387 (2018)
Zhang, Z., Luo, P., Loy, C.C., Tang, X.: Facial landmark detection by deep multi-task learning. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 94–108. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_7
Ye, J., Li, G., Chen, D., Yang, H., Zhe, S., Xu, Z.: Block-term tensor neural networks. Neural Netw. 11–21(130), 0893–6080 (2020)
Long, M., Cao, Z., Wang, J., Philip, S.Y.: Learning multiple tasks with multi linear relationship networks. In: NIPS, pp. 1594–1603 (2017)
Liu, B., et al.: Learning from semantically dependent multi-tasks. In: IJCNN, pp. 3498–3505. IEEE (2017)
Ren, Y., Yan, X., Hu, Z., Xu, Z.: Self-paced multi-task multi-view capped-norm clustering. In: Cheng, L., Leung, A.C.S., Ozawa, S. (eds.) ICONIP 2018. LNCS, vol. 11304, pp. 205–217. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04212-7_18
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, M., Su, Z., Luo, X., Pan, Y., Zheng, S., Xu, Z. (2020). Concatenated Tensor Networks for Deep Multi-Task Learning. In: Yang, H., Pasupa, K., Leung, A.CS., Kwok, J.T., Chan, J.H., King, I. (eds) Neural Information Processing. ICONIP 2020. Communications in Computer and Information Science, vol 1333. Springer, Cham. https://doi.org/10.1007/978-3-030-63823-8_59
Download citation
DOI: https://doi.org/10.1007/978-3-030-63823-8_59
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63822-1
Online ISBN: 978-3-030-63823-8
eBook Packages: Computer ScienceComputer Science (R0)