Abstract
Gestures during spoken dialog play a central role in human communication. As a consequence, models of gesture generation are a key challenge in research on virtual humans, embodied agents capable of face-to-face interaction with people. Machine learning approaches to gesture generation must take into account the conceptual content in utterances, physical properties of speech signals and the physical properties of the gestures themselves. To address this challenge, we proposed a gestural sign scheme to facilitate supervised learning and presented the DCNF model, a model to jointly learn deep neural networks and second order linear chain temporal contingency. The approach we took realizes both the mapping relation between speech and gestures while taking account temporal relations among gestures. Our experiments on human co-verbal dataset shows significant improvement over previous work on gesture prediction. A generalization experiment performed on handwriting recognition also shows that DCNFs outperform the state-of-the-art approaches.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
We have experimented with both the logistic and the rectified linear (\(\max (a\theta ^{w}, 0)\)) functions with similar results. Because of space constraints, we are focusing on the logistic function.
- 2.
The full derivation of the gradient was omitted because of space constraint.
References
Bergmann, K., Kahl, S., Kopp, S.: Modeling the semantic coordination of speech and gesture under cognitive and linguistic constraints. In: Aylett, R., Krenn, B., Pelachaud, C., Shimodaira, H. (eds.) IVA 2013. LNCS, vol. 8108, pp. 203–216. Springer, Heidelberg (2013)
Bird, S., Loper, E., Klein, E.: Natural Language Processing with Python. OReilly Media Inc, Santa Clara (2009)
Brugman, H., Russel, A., Nijmegen, X.: Annotating multi-media/multimodal resources with ELAN. In: Proceedings of the Fourth International Conference on Language Resources and Evaluation, LREC 2004, pp. 2065–2068 (2004)
Calbris, G.: Elements of Meaning in Gesture: Gesture Studies 5. John Benjamins, Philadelphia (2011)
Cassell, J., Prevost, S.: Distribution of semantic features across speech and gesture by humans and computers. In: Workshop on the Integration of Gesture in Language and Speech (1996)
Cassell, J.: Embodied conversational interface agents. Commun. ACM 43(4), 70–78 (2000)
Cassell, J., Vilhjálmsson, H.H., Bickmore, T.: Beat: the behavior expression animation toolkit. In: SIGGRAPH 2001 Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 477–486. ACM, New York (2001)
Chiu, C.-C., Marsella, S.: How to train your avatar: a data driven approach to gesture generation. In: Vilhjálmsson, H.H., Kopp, S., Marsella, S., Thórisson, K.R. (eds.) IVA 2011. LNCS, vol. 6895, pp. 127–140. Springer, Heidelberg (2011)
Chiu, C.C., Marsella, S.: Gesture generation with low-dimensional embeddings. In: Proceedings of the 13th International Joint Conference on Autonomous Agents and Multiagent Systems. AAMAS 2013 (2014)
Do, T., Artieres, T.: Neural conditional random fields. In: International Conference on Artificial Intelligence and Statistics (AI-STATS), pp. 177–184 (2010)
Ennis, C., McDonnell, R., O’Sullivan, C.: Seeing is believing: body motion dominates in multisensory conversations. In: ACM SIGGRAPH 2010 papers, SIGGRAPH 2010, pp. 91:1–91:9. ACM, New York(2010)
Ennis, C., O’Sullivan, C.: Perceptually plausible formations for virtual conversers. Comput. Animation Virtual Worlds 23(3–4), 321–329 (2012)
Fujii, Y., Yamamoto, K., Nakagawa, S.: Deep-hidden conditional neural fields for continuous phoneme speech recognition. In: International Workshop of Statistical Machine Learning for Speech (IWSML) (2012)
Goldin-Meadow, S., Alibali, M.W., Church, R.B.: Transitions in concept acquisition: using the hand to read the mind. Psychol. Rev. 100(2), 279–297 (1993)
Gratch, J., Artstein, R., Lucas, G., Stratou, G., Scherer, S., Nazarian, A., Wood, R., Boberg, J., Devault, D., Marsella, S., Traum, D., Rizzo, A.S., Morency, L.P.: The distress analysis interview corpus of human and computer interviews. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014), European Language Resources Association (ELRA), Reykjavik, Iceland, May 2014
Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) (2013)
Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Improving neural networks by preventing co-adaptation of feature detectors (2012). pre-print arXiv:1207.0580v1
Kipp, M.: Gesture generation by imitation - from human behavior to computer character animation. Ph.D. thesis, Saarland University (2004)
Kopp, S., Bergmann, K.: Individualized gesture production in embodied conversational agents. In: Zacarias, M., de Oliveira, J.V. (eds.) Human-Computer Interaction. SCI, vol. 396, pp. 287–302. Springer, Heidelberg (2012)
Lafferty, J.D., McCallum, A., Pereira, F.C.N.: Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: ICML, pp. 282–289 (2001)
Lee, J., Marsella, S.C.: Nonverbal behavior generator for embodied conversational agents. In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 243–255. Springer, Heidelberg (2006)
Levine, S., Krähenbühl, P., Thrun, S., Koltun, V.: Gesture controllers. In: ACM SIGGRAPH 2010 papers, pp. 124:1–124:11. ACM, New York (2010)
Levine, S., Theobalt, C., Koltun, V.: Real-time prosody-driven synthesis of body language. ACM Trans. Graph. 28, 172:1–172:10 (2009). http://doi.acm.org/10.1145/1618452.1618518
Marsella, S.C., Xu, Y., Lhommet, M., Feng, A.W., Scherer, S., Shapiro, A.: Virtual character performance from speech. In: Symposium on Computer Animation. Anaheim, CA, July 2013
McNeill, D.: So you think gestures are nonverbal? Psychol. Rev. 92(3), 350–371 (1985)
Mohamed, A.R., Dahl, G.E., Hinton, G.: Acoustic modeling using deep belief networks. IEEE Trans. Audio Speech Lang. Process. 20(1), 14–22 (2012)
Neff, M., Kipp, M., Albrecht, I., Seidel, H.P.: Gesture modeling and animation based on a probabilistic re-creation of speaker style. ACM Trans. Graph. 27(1), 1–24 (2008)
Peng, J., Bo, L., Xu, J.: Conditional neural fields. In: NIPS, pp. 1419–1427 (2009)
Rickel, J., Johnson, W.L.: Task-oriented collaboration with embodied agents in virtual worlds. In: Cassell, J., Sullivan, J., Prevost, S. (eds.) Embodied Conversational Agents, pp. 95–122. MIT Press, Cambridge (2000)
Salem, M., Rohlfing, K.J., Kopp, S., Joublin, F.: A friendly gesture: investigating the effect of multimodal robot behavior in human-robot interaction. In: 2011 IEEE RO-MAN, pp. 247–252, July 2011
Scherer, S., Kane, J., Gobl, C., Schwenker, F.: Investigating fuzzy-input fuzzy-output support vector machines for robust voice quality classification. Comput. Speech Lang. 27(1), 263–287 (2013)
Stone, M., DeCarlo, D., Oh, I., Rodriguez, C., Stere, A., Lees, A., Bregler, C.: Speaking with hands: creating animated conversational characters from recordings of human performance. In: ACM SIGGRAPH 2004 Papers, SIGGRAPH 2004, pp. 506–513. ACM, New York (2004)
Sutskever, I., Martens, J., Hinton, G.: Generating text with recurrent neural networks. In: ICML (2011)
Taskar, B., Guestrin, C., Koller, D.: Max-margin Markov networks. In: Thrun, S., Saul, L., Schölkopf, B. (eds.) Advances in Neural Information Processing Systems 16. MIT Press, Cambridge (2004)
Taylor, G., Hinton, G.: Factored conditional restricted Boltzmann machines for modeling motion style. In: Bottou, L., Littman, M. (eds.) Proceedings of the 26th International Conference on Machine Learning, pp. 1025–1032. Omnipress, Montreal, June 2009
Wagner, P., Malisz, Z., Kopp, S.: Gesture and speech in interaction: an overview. Speech Commun. 57, 209–232 (2014)
Weiss, D., Sapp, B., Taskar, B.: Structured prediction cascades (2012). preprint arXiv:1208.3279v1
Yu, D., Deng, L., Wang, S.: Learning in the deep-structured conditional random fields. In: NIPS Workshop on Deep Learning for Speech Recognition and Related Applications (2009)
Acknowledgements
The projects or effort described here has been sponsored by the U.S. Army. Any opinions, content or information presented does not necessarily reflect the position or the policy of the United States Government, and no official endorsement should be inferred.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Chiu, CC., Morency, LP., Marsella, S. (2015). Predicting Co-verbal Gestures: A Deep and Temporal Modeling Approach. In: Brinkman, WP., Broekens, J., Heylen, D. (eds) Intelligent Virtual Agents. IVA 2015. Lecture Notes in Computer Science(), vol 9238. Springer, Cham. https://doi.org/10.1007/978-3-319-21996-7_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-21996-7_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-21995-0
Online ISBN: 978-3-319-21996-7
eBook Packages: Computer ScienceComputer Science (R0)