Abstract
To understand speakers’ attitudes and intentions in real Voice Dialogue Applications (VDAs), effective emphasis inference from users’ queries may play an important role. However, in VDAs, there are tremendous amount of uncertain speakers with a great diversity of users’ dialects, expression preferences, which challenge the traditional emphasis detection methods. In this paper, to better infer emphasis for real voice data, we propose an attentive multimodal neural network. Specifically, first, beside the acoustic features, extensive textual features are applied in modelling. Then, considering the feature in-dependency, we model the multi-modal features utilizing a Multi-path convolutional neural network (MCNN). Furthermore, combining high-level multi-modal features, we train an emphasis classifier by attending on the textual features with an attention-based bidirectional long short-term memory network (ABLSTM), to comprehensively learn discriminative features from diverse users. Our experimental study based on a real-world dataset collected from Sogou Voice Assistant (https://yy.sogou.com/) show that our method outperforms (over 1.0–15.5% in terms of F1 measure) alternative baselines.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Brenier, J.M., Cer, D.M., Jurafsky, D.: The detection of emphatic words using acoustic and lexical features. In: Ninth European Conference on Speech Communication and Technology (2005)
Cernak, M., Honnet, P.E.: An empirical model of emphatic word detection. In: Interspeech, pp. 573–577 (2015)
Chen, J.Y., Lan, W.: Automatic lexical stress detection for Chinese learners’ of English. In: International Symposium on Chinese Spoken Language Processing (2011)
Chen, Y., Yuan, J., You, Q., Luo, J.: Twitter sentiment analysis via bi-sense emoji embedding and attention-based LSTM. arXiv preprint arXiv:1807.07961 (2018)
Do, Q.T., Takamichi, S., Sakti, S., Neubig, G., Toda, T., Nakamura, S.: Preserving word-level emphasis in speech-to-speech translation using linear regression HSMMs. In: Sixteenth Annual Conference of the International Speech Communication Association (2015)
Do, Q.T., Toda, T., Neubig, G., Sakti, S., Nakamura, S.: Preserving word-level emphasis in speech-to-speech translation. IEEE/ACM Trans. Audio Speech Lang. Process. 25(3), 544–556 (2017)
Fan, Y., Qian, Y., Xie, F.L., Soong, F.K.: TTS synthesis with bidirectional LSTM based recurrent neural networks. In: Fifteenth Annual Conference of the International Speech Communication Association (2014)
Ferrer, L., Bratt, H., Richey, C., Franco, H., Abrash, V., Precoda, K.: Lexical stress classification for language learning using spectral and segmental features. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7704–7708 (2014)
Gers, F.A., Schraudolph, N.N., Schmidhuber, J.: Learning precise timing with LSTM recurrent networks. J. Mach. Learn. Res. 3(Aug), 115–143 (2002)
Heldner, M.: Spectral emphasis as an additional source of information in accent detection. In: ISCA Tutorial and Research Workshop (ITRW) on Prosody in Speech Recognition and Understanding (2001)
Kennedy, L.S., Ellis, D.P.W.: Pitch-based emphasis detection for characterization of meeting recordings. In: 2003 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2003, pp. 243–248 (2003)
Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Ladd, D.R., Morton, R.: The perception of intonational emphasis: continuous or categorical? J. Phon. 25(3), 313–342 (1997)
Li, L., Wu, Z., Xu, M., Meng, H.M., Cai, L.: Combining CNN and BLSTM to extract textual and acoustic features for recognizing stances in mandarin ideological debate competition. In: Interspeech, pp. 1392–1396 (2016)
Li, Z., Sun, M.: Punctuation as implicit annotations for Chinese word segmentation. Comput. Linguist. 35(4), 505–512 (2009)
Ning, Y., et al.: Learning cross-lingual knowledge with multilingual BLSTM for emphasis detection with limited training data. In: ICASSP 2017–2017 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5615–5619 (2017)
Schnall, A., Heckmann, M.: Integrating sequence information in the audio-visual detection of word prominence in a human-machine interaction scenario. In: Fifteenth Annual Conference of the International Speech Communication Association (2014)
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Sig. Process. 45(11), 2673–2681 (1997)
Zhang, L., et al.: Emphasis detection for voice dialogue applications using multi-channel convolutional bidirectional long short-term memory network. In: 2018 11th International Symposium on Chinese Spoken Language Processing (ISCSLP), pp. 210–214. IEEE (2018)
Zhou, S., Jia, J., Wang, Q., Dong, Y., Yin, Y., Lei, K.: Inferring emotion from conversational voice data: a semi-supervised multi-path generative neural network approach. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
Acknowledgements
This work is supported by Tiangong Institute for Intelligent Computing, Tsinghua University and the state key program of the National Natural Science Foundation of China (NSFC) (No. 61831022).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, S. et al. (2020). Inferring Emphasis for Real Voice Data: An Attentive Multimodal Neural Network Approach. In: Ro, Y., et al. MultiMedia Modeling. MMM 2020. Lecture Notes in Computer Science(), vol 11962. Springer, Cham. https://doi.org/10.1007/978-3-030-37734-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-37734-2_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37733-5
Online ISBN: 978-3-030-37734-2
eBook Packages: Computer ScienceComputer Science (R0)