Abstract
Temporal information prevails in multimodal sequence data, such as video data and speech signals. In this paper, we propose a two-stage learning to model the temporal information in multimodal sequences. At the first learning stage, static representative features are extracted from each modality at every time step. Then joint representations across various modalities are effectively learned within a joint fusion layer. The second one is to transfer the static features into corresponding dynamical features by jointly learning the temporal information and dependencies between different time steps with a Long Short-Term Memory (LSTM). Compared with previous multimodal methods, the proposed model is efficient in learning temporal joint representations. Evaluated on Big Bang Theory speaker recognition dataset and AVLetters speech recognition dataset, our model proves to outperform other methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amer, M.R., Siddiquie, B., Khan, S., Divakaran, A., Sawhney, H.: Multimodal fusion using dynamic hybrid models. In: IEEE Winter Conference on Applications of Computer Vision, pp. 556–563, March 2014
Bauml, M., Tapaswi, M., Stiefelhagen, R.: Semi-supervised learning with constraints for person identification in multimedia data. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013
Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Hong, C., Yu, J., Wan, J., Tao, D., Wang, M.: Multimodal deep autoencoder for human pose recovery. IEEE Trans. Image Process. 24(12), 5659–5670 (2015)
Hu, D., Li, X., Lu, X.: Temporal multimodal learning in audiovisual speech recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
Hu, Y., Ren, J.S., Dai, J., Yuan, C., Xu, L., Wang, W.: Deep multimodal speaker naming. In: Proceedings of the 23rd ACM International Conference on Multimedia, MM 2015, NY, USA, pp. 1107–1110 (2015). doi:10.1145/2733373.2806293
Huang, J., Kingsbury, B.: Audio-visual deep learning for noise robust speech recognition. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7596–7599 (2013)
Jamil, M., Rahman, G.R.S.: Speaker identification using Mel frequency cepstral coefficients (2004)
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28–July 2, pp. 689–696 (2011)
Ren, J., Hu, Y., Tai, Y.W., Wang, C., Xu, L., Sun, W., Yan, Q.: Look, listen and learn – a multimodal LSTM for speaker identification. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI 2016, pp. 3581–3587. AAAI Press (2016)
Sohn, K., Shang, W., Lee, H.: Improved multimodal deep learning with variation of information. In: International Conference on Neural Information Processing Systems, pp. 2141–2149 (2014)
Srivastava, N., Salakhutdinov, R.: Multimodal learning with deep boltzmann machines. J. Mach. Learn. Res. 15, 2949–2980 (2014). http://jmlr.org/papers/v15/srivastava14b.html
Stiefelhagen, R.: “Knock! knock! who is it?” probabilistic person identification in TV-series. In: Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR 2012, pp. 2658–2665 (2012). http://dl.acm.org/citation.cfm?id=2354409.2354974
Acknowledgment
This work is supported by the National Natural Science Foundation of China (Grant No. 61502174, 61402181), the Natural Science Foundation of Guangdong Province (Grant No. S2012010009961, 2015A030313215), the Science and Technology Planning Project of Guangdong Province (Grant No. 2016A040403046), the Guangzhou Science and Technology Planning Project (Grant No. 201704030051, 2014J4100006), the Opening Project of Guangdong Province Key Laboratory of Big Data Analysis and Processing (Grant No. 2017014), and the Fundamental Research Funds for the Central Universities (Grant No. D2153950).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Ma, Q., Shen, L., Su, R., Chen, J. (2017). Two-Stage Temporal Multimodal Learning for Speaker and Speech Recognition. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, ES. (eds) Neural Information Processing. ICONIP 2017. Lecture Notes in Computer Science(), vol 10635. Springer, Cham. https://doi.org/10.1007/978-3-319-70096-0_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-70096-0_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70095-3
Online ISBN: 978-3-319-70096-0
eBook Packages: Computer ScienceComputer Science (R0)