Abstract
Multi-modal deep learning has achieved great success in many applications. Previous works are mostly based on auto-encoder networks or paired networks, however, these methods generally consider the consensus principle on the output layers and always need deep structures. In this paper, we propose a novel Cascade Deep Multi-Modal network structure (CDMM), which generates deep multi-modal networks with a cascade structure by maximizing the correlations between each hidden homogeneous layers. In CDMM, we simultaneously train two nonlinear mappings layer by layer, and the consistency between different modal output features is considered in each homogeneous layer, besides, the representation learning ability can be forward enhanced by considering the raw feature representation simultaneously for each layer. Finally, experiments on 5 real-world datasets validate the effectiveness of our method.
This work was supported by the National Key R&D Program of China (2018YFB1004300), NSFC (61773198, 61632004).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Akaho, S.: A kernel method for canonical correlation analysis, pp. 263–269 (2007)
Andrew, G., Arora, R., Bilmes, J.A., Livescu, K.: Deep canonical correlation analysis. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, pp. 1247–1255 (2013)
Arora, R., Mianjy, P., Marinov, T.V.: Stochastic optimization for multiview representation learning using partial least squares. In: Proceedings of the 33rd International Conference on Machine Learning, New York, NY, pp. 4847–4855 (2016)
Hardoon, D.R., Szedmak, S.R., Shawe-Taylor, J.R.: Canonical Correlation Analysis: An Overview with Application to Learning Methods. MIT Press, Cambridge (2004)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Hodosh, M., Young, P., Hockenmaier, J.: Framing image description as a ranking task: data, models and evaluation metrics. JAIR 47, 853–899 (2013)
Hotelling, H.: Relations between two sets of variates. Biometrika 28(3–4), 321–377 (1936)
Hu, D., Li, X., Lu, X.: Temporal multimodal learning in audiovisual speech recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, pp. 3574–3582 (2016)
Kan, M., Shan, S., Chen, X.: Multi-view deep network for cross-view classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, pp. 4847–4855 (2016)
Kang, G., Li, J., Tao, D.: Shakeout: a new regularized deep neural network training scheme. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix, Arizona, pp. 1751–1757 (2016)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: Proceedings of the 28th International Conference on Machine Learning, Bellevue, Washington, pp. 689–696 (2011)
Rasiwasia, N., Pereira, J.C., Coviello, E., Doyle, G., Lanckriet, G.R., Levy, R., Vasconcelos, N.: A new approach to cross-modal multimedia retrieval. In: Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy, pp. 251–260 (2010)
Rupnik, J., Shawe-Taylor, J.: Multi-view canonical correlation analysis. In: Slovenian KDD Conference on Data Mining and Data Warehouses, Ljubljana, Yugoslavia, pp. 1–4 (2010)
Shrivastava, A., Rastegari, M., Shekhar, S., Chellappa, R., Davis, L.S.: Class consistent multi-modal fusion with binary features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, pp. 2282–2291 (2015)
Tian, F., Gao, B., Cui, Q., Chen, E., Liu, T.Y.: Learning deep representations for graph clustering. In: Proceedings of the 28th AAAI Conference on Artificial Intelligence, Quebec, Canada, pp. 1293–1299 (2014)
Wang, W., Arora, R., Livescu, K., Bilmes, J.: On deep multi-view representation learning. In: Proceedings of the 32nd International Conference on Machine Learning, Lille, France, pp. 1083–1092 (2015)
Yang, Y., Zhan, D.C., Jiang, Y.: Deep learning for fixed model reuse. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence, New York, NY, pp. 1033–1039 (2017)
Zhou, Z.H., Feng, J.: Deep forest: towards an alternative to deep neural networks. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia (2017)
Zhu, X., Huang, Z., Wu, X.: Multi-view visual classification via a mixed-norm regularizer. In: Pei, J., Tseng, V.S., Cao, L., Motoda, H., Xu, G. (eds.) PAKDD 2013, Part I. LNCS (LNAI), vol. 7818, pp. 520–531. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37453-1_43
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Yang, Y., Wu, YF., Zhan, DC., Jiang, Y. (2018). Deep Multi-modal Learning with Cascade Consensus. In: Geng, X., Kang, BH. (eds) PRICAI 2018: Trends in Artificial Intelligence. PRICAI 2018. Lecture Notes in Computer Science(), vol 11013. Springer, Cham. https://doi.org/10.1007/978-3-319-97310-4_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-97310-4_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-97309-8
Online ISBN: 978-3-319-97310-4
eBook Packages: Computer ScienceComputer Science (R0)