Abstract
To engage in natural interactions with humans, social robots should produce speech-accompanying non-verbal behaviors such as hand and arm gestures. Given the special constraints imposed by the physical properties of a humanoid robot, successful multimodal synchronization is difficult to achieve. Introducing the first closed-loop approach to speech and gesture generation for humanoid robots, we propose a multimodal scheduler for improved synchronization based on two novel features, namely an experimentally fitted forward model and a feedback-based adaptation mechanism. Technical results obtained with the implemented scheduler demonstrate the feasibility of our approach; empirical results from an evaluation study highlight the implications of the present work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bremner, P., Pipe, A., Melhuish, C., Fraser, M., Subramanian, S.: Conversational Gestures in Human-Robot Interaction. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pp. 1645–1649 (2009)
Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., Yan, H.: Human Conversation as a System Framework: Desigining Embodied Conversational Agents. In: Embodied Conversational Agents, pp. 29–63. MIT Press, Cambridge (2000)
Cassell, J., Vilhjálmsson, H., Bickmore, T.: BEAT: the Behavior Expression Animation Toolkit. In: Proceedings of ACM SIGGRAPH 2001 (2001)
Desmurget, M., Grafton, S.: Forward modeling allows feedback control for fast reaching movements. Trends in Cognitive Sciences 4(11), 423–431 (2000)
Gienger, M., Bolder, B., Dunn, M., Sugiura, H., Janssen, H., Goerick, C.: Predictive Behavior Generation – A Sensor-Based Walking and Reaching Architecture for Humanoid Robots. In: Autonome Mobile Systeme 2007, pp. 275–281 (2007)
Gienger, M., Janßen, H., Goerick, S.: Task-Oriented Whole Body Motion for Humanoid Robots. In: Proceedings of the IEEE-RAS International Conference on Humanoid Robots, Tsukuba, Japan (2005)
Kendon, A.: Gesture: Visible Action as Utterance. Gesture 6(1), 119–144 (2004)
Kopp, S., Wachsmuth, I.: Synthesizing Multimodal Utterances for Conversational Agents. Computer Animation and Virtual Worlds 15(1), 39–52 (2004)
Kranstedt, A., Kopp, S., Wachsmuth, I.: MURML: A Multimodal Utterance Representation Markup Language for Conversational Agents. In: AAMAS 2002 Workshop on Embodied Conversational Agents - Let’s Specify and Evaluate Them (2002)
Le, Q.A., Hanoune, S., Pelachaud, C.: Design and implementation of an expressive gesture model for a humanoid robot. In: Proceedings of the 11th IEEE-RAS International Conference on Humanoid Robots, Bled, Slovenia, pp. 134–140 (2011)
McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)
McNeill, D.: Gesture and Thought. University of Chicago Press, Chicago (2005)
Mead, R., Wade, E., Johnson, P., St. Clair, A., Chen, S., Matarić, M.: An Architecture for Rehabilitation Task Practice in Socially Assistive Human-Robot Interaction. In: Proceedings of IEEE RO-MAN 2010, pp. 404–409 (2010)
Ng-Thow-Hing, V., Luo, P., Okita, S.: Synchronized Gesture and Speech Production for Humanoid Robots. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4617–4624 (2010)
Niewiadomski, R., Bevacqua, E., Mancini, M., Pelachaud, C.: Greta: An Interactive Expressive ECA System. In: Proceedings of 8th International Conference on Autonomous Agents and Multiagent Systems, pp. 1399–1400 (2009)
Salem, M.: Conceptual Motorics – Generation and Evaluation of Communicative Robot Gesture. Dissertation. Logos Verlag, Berlin (2012)
Schröder, M., Trouvain, J.: The German Text-to-Speech Synthesis System MARY: A Tool for Research, Development and Teaching. International Journal of Speech Technology, 365–377 (2003)
Sidner, C., Lee, C., Lesh, N.: The Role of Dialog in Human Robot Interaction. In: International Workshop on Language Understanding and Agents for Real World Interaction (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Salem, M., Kopp, S., Joublin, F. (2013). Closing the Loop: Towards Tightly Synchronized Robot Gesture and Speech. In: Herrmann, G., Pearson, M.J., Lenz, A., Bremner, P., Spiers, A., Leonards, U. (eds) Social Robotics. ICSR 2013. Lecture Notes in Computer Science(), vol 8239. Springer, Cham. https://doi.org/10.1007/978-3-319-02675-6_38
Download citation
DOI: https://doi.org/10.1007/978-3-319-02675-6_38
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-02674-9
Online ISBN: 978-3-319-02675-6
eBook Packages: Computer ScienceComputer Science (R0)