Abstract
Multi-task learning (MTL) is a promising research field of machine learning, in which the training process of the neural network is equivalent to multi-objective optimization. On one hand, MTL trains all the network weights simultaneously to converge the multi-task loss. On the other hand, multi-objective optimization aims to find the optimum solution, which satisfies the constraints and optimizes the vector of objective functions. Therefore, the performance of MTL is dominated by the computation of the multi-objective solution. This paper proposes a method based on Riemannian optimization to solve the multi-objective optimization in MTL. Firstly, multi-objective optimization is reduced to its Karush-Kuhn-Tucker (KKT) condition as the optimum solution of constrained quadratic optimization. Secondly, by mapping the Euclidean space of the constraint into manifold, the quadratic optimization is transformed to an unconstrained problem. Finally, Riemannian optimization algorithm is used to compute the solution of this problem, which gives a Pareto direction towards the KKT condition. We perform experiments on the MultiMNIST and Fashion MNIST datasets, and the experimental results demonstrate the efficiency of our method.
This work was supported by the Fundamental Research Funds for the Central Universities of China under Grant No. 20720190028.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Caruana, R.: Multitask learning. In: Thrun, S., Pratt, L. (eds.) Learning to learn. Springer, Boston (1998). https://doi.org/10.1007/978-1-4615-5529-2_5
Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2650–2658 (2015)
Kokkinos, I.: UberNet: training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. arXiv:1609.02132 (2017)
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: OverFeat: integrated recognition, localization and detection using convolutional networks. arXiv:1312.6229 (2013)
Désidéri, J.-A.: Multiple-gradient descent algorithm (MGDA) for multiobjective optimization. C.R. Math. 350(5), 313–318 (2012)
Sener, O., Koltun, V.: Multi-task learning as multi-objective optimization. arXiv:1810.04650 (2018)
Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. arXiv:1710.09819 (2017)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747 (2017)
Ruder, S.: An overview of multi-task learning in deep neural networks. arXiv:1706.05098 (2017)
Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)
MartÃ, M., Maki A.: A multitask deep learning model for real-time deployment in embedded systems. arXiv:1711.00146 (2017)
Bilen, H., Vedaldi, A.: Integrated perception with recurrent multi-task neural networks. arXiv:1606.01735 (2016)
Collobert, R., Weston, J.: A unified architecture for natural language processing: deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine Learning, pp. 160–167 (2008)
Huang, J.-T., Li, J., Yu, D., Deng, L., Gong, Y.: Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7304–7308 (2013)
Huang, Z., Li, J., Siniscalchi, S.M., Chen, I.-F., Wu, J., Lee, C.-H.: Rapid adaptation for deep neural networks through multi-task learning. In: Proceedings of the Annual Conference of the International Speech Communication Association, pp. 3625–3629 (2015)
Baxter, J.: A model of inductive bias learning. J. Artif. Intell. Res. 12, 149–198 (2000)
Li, C., Georgiopoulos, M., Anagnostopoulos, G.C.: Pareto-path multi-task multiple kernel learning. arXiv:1404.3190 (2014)
Teichmann, M., Weber, M., Zoellner, M., Cipolla, R., Urtasun, R.: MultiNet: real-time joint semantic reasoning for autonomous driving. arXiv:1612.07695 (2016)
Zhou, D., Wang, J., Jiang, B., Guo, H., Li, Y.: Multi-task multi-view learning based on cooperative multi-objective optimization. IEEE Access 6, 19465–19477 (2017)
Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482–7491 (2018)
Miettinen, K.: Nonlinear Multiobjective Optimization. Springer, New York (2012). https://doi.org/10.1007/978-1-4615-5563-6
Ehrgott, M.: Multicriteria Optimization. Springer, Heidelberg (2005). https://doi.org/10.1007/3-540-27659-9
Zhou, A., Zhang, Q., Jin, Y.: Approximating the set of pareto-optimal solutions in both the decision and objective spaces by an estimation of distribution algorithm. IEEE Trans. Evol. Comput. 13(5), 1167–1189 (2009)
Mukhopadhyay, A., Bandyopadhyay, S., Maulik, U.: Multi-class clustering of cancer subtypes through SVM based ensemble of pareto-optimal solutions for gene marker identification. PLoS ONE 5(11), e13803 (2010)
Suga, K., Kato, S., Hiyama, K.: Structural analysis of Pareto-optimal solution sets for multi-objective optimization: an application to outer window design problems using multiple objective genetic algorithms. Build. Environ. 45(5), 1144–1152 (2010)
Poirion, F., Mercier, Q., Désidéri, J.-A.: Descent algorithm for nonsmooth stochastic multiobjective optimization. Comput. Optim. Appl. 68(2), 317–331 (2017). https://doi.org/10.1007/s10589-017-9921-x
Peitz, S., Dellnitz, M.: Gradient-based multiobjective optimization with uncertainties. In: Maldonado, Y., Trujillo, L., Schütze, O., Riccardi, A., Vasile, M. (eds.) NEO 2016. SCI, vol. 731, pp. 159–182. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-64063-1_7
Boumal, N.: An introduction to optimization on smooth manifolds (2020)
Project webpage. https://github.com/NicolasBoumal/manopt
Alimisis, F., Orvieto, A., Bécigneul, G., Lucchi, A.: A continuous-time perspective for modeling acceleration in Riemannian optimization. arXiv:1910.10782 (2019)
Bonnabel, S.: Stochastic gradient descent on Riemannian manifolds. IEEE Trans. Autom. Control 58(9), 2217–2229 (2013)
Sun, Y., Gao, J., Hong, X., Mishra, B., Yin, B.: Heterogeneous tensor decomposition for clustering via manifold optimization. IEEE Trans. Pattern Anal. Mach. Intell. 38(3), 476–489 (2015)
Kuhn, H.W., Tucker, A.W.: Nonlinear programming. In: Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability (1951)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Cai, T., Song, L., Li, G., Liao, M. (2021). Multi-task Learning with Riemannian Optimization. In: Huang, DS., Jo, KH., Li, J., Gribova, V., Hussain, A. (eds) Intelligent Computing Theories and Application. ICIC 2021. Lecture Notes in Computer Science(), vol 12837. Springer, Cham. https://doi.org/10.1007/978-3-030-84529-2_42
Download citation
DOI: https://doi.org/10.1007/978-3-030-84529-2_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-84528-5
Online ISBN: 978-3-030-84529-2
eBook Packages: Computer ScienceComputer Science (R0)