Article

Cross-modal coordination of expressive strength between voice and gesture for personified media

Authors:

Tomoko Yonezawa,

Kiyoshi KogureAuthors Info & Claims

ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

Pages 43 - 50

https://doi.org/10.1145/1180995.1181006

Published: 02 November 2006 Publication History

Abstract

The aim of this paper is to clarify the relationship between the expressive strengths of gestures and voice for embodied and personified interfaces. We conduct perceptual tests using a puppet interface, while controlling singing-voice expressions, to empirically determine the naturalness and strength of various combinations of gesture and voice. The results show that (1) the strength of cross-modal perception is affected more by gestural expression than by the expressions of a singing voice, and (2) the appropriateness of cross-modal perception is affected by expressive combinations between singing voice and gestures in personified expressions. As a promising solution, we propose balancing a singing voice and gestural expressions by expanding and correcting the width and shape of the curve of expressive strength in the singing voice.

References

[1]

J. Bates. The role of emotion in believable agents. Communications of the ACM, pages 122--125, 1994.

Digital Library

[2]

T. W. Bickmore and J. Cassell. Small talk and conversational storytelling in embodied conversational interface agent. AAAI fall symposium on narrative intelligence, pages 87--92, 1999.

[3]

J. Cassell, J. Sullivan, S. Prevost, and E. Churchill. Embodied Conversational Agents. MIT Press, 2000.

Digital Library

[4]

T. Chen and R. R. Rao. Audio-visual integration in multimodal communication. Proc. IEEE, 86(5):837--852, 1998.

[5]

B. Duffy. Anthropomorphism and the social robot. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2002.

[6]

M. Fujita and K. Kageyama. An open architecture for robot entertainment. Proc. the First International Conference on Autonomous Agents, pages 435--442, 1997.

Digital Library

[7]

M. Imai, T. Ono, and T. Etani. Attractive interface for human robot interaction. Proc. of 8th IEEE International Workshop on Robot and Human Communication (ROMAN'99), pages 124--129, 1999.

[8]

S. Iwamiya. Multimodal communication by music and motion picture. Proc. of 7th International Conference on Music Perception and Cognition, pages 3--8, 2002.

[9]

H. Kawahara, I. Masuda-Kasuse, and A. Cheveigne. Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based f0 extraction: Possible role of a reptitive structure in sounds. Speech Communication, 27:187--207, 1999.

Digital Library

[10]

H. Kawahara and H. Matsui. Auditory morphing based on an elastic perceptual distance metric in an interference-free time-frequency representation. Proc. ICASSP '2003, I:256--259, 2003.

[11]

H. Mcgurk and M. Lewis. Space perception in early infancy: perception within a common auditory-visual space. Science, 186:649--650, 1974.

[12]

S. M. Puckette. "pure data". Proc. ICMC 1997, pages 224--227, 1997.

[13]

D. Sekiguchi, M. Inami, and S. Tachi. Robotphone: Rui for interpersonal communication. CHI2001 Extended Abstracts, pages 277--278, 2001.

Digital Library

[14]

Y. Sogabe, K. Kakehi, and H. Kawahara. Psychological evaluation of emotional speech using a new morphing method. 4th ICCS International Conference on Cognitive Science, 2003.

[15]

M. Yamamoto and T. Watanabe. Timing control effects of utterance to communicative actions on embodied interaction with a robot. Proc. IEEE Workshop on Robot and Human Interactive Communication, pages 467--472, 2004.

[16]

T. Yonezawa and K. Mase. Musically expressive doll in face-to-face communication. IEEE Proc. International Conference of Multimodal Interfaces, pages 417--422, 2002.

Digital Library

[17]

T. Yonezawa, N. Suzuki, K. Mase, and K. Kogure. Gradually changing expression of singing voice based on morphing. Proc. Interspeech 2005, pages 541--544, 2005.

[18]

T. Yonezawa, N. Suzuki, K. Mase, and K. Kogure. Handysinger: Expressive singing voice morphing using personified hand-puppet interface. Proc. NIME2005, pages 121--126, 2005.

Digital Library

Cited By

Yonezawa TYamazoe HUtsumi AAbe S(2008)GazeRoboard: Gaze-communicative guide system in daily life on stuffed-toy robot with interactive display board2008 IEEE/RSJ International Conference on Intelligent Robots and Systems10.1109/IROS.2008.4650692(1204-1209)Online publication date: Sep-2008
https://doi.org/10.1109/IROS.2008.4650692
Yonezawa TYamazoe HUtsumi AAbe SMase KMassaro DTakeda KRoy DPotamianos A(2007)Gaze-communicative behavior of stuffed-toy robot with joint attention and eye contact based on ambient gaze-trackingProceedings of the 9th international conference on Multimodal interfaces10.1145/1322192.1322218(140-145)Online publication date: 12-Nov-2007
https://dl.acm.org/doi/10.1145/1322192.1322218

Index Terms

Cross-modal coordination of expressive strength between voice and gesture for personified media
1. Information systems
  1. Information systems applications
    1. Multimedia information systems

Recommendations

Recognition of the Emotional State of Children by Video and Audio Modalities by Indian and Russian Experts
Speech and Computer
Abstract
The paper presents the results of perceptual experiment by Indian and Russian experts for recognition of the emotional states of Russian children by video and audio modalities. The participants of the study were 64 Russian children (32 – boys & 32 ...
A multimodal shared network with a cross-modal distribution constraint for continuous emotion recognition
Abstract
Continuous emotion recognition has been a compelling topic in affective computing because it can interpret human emotions subtly and continuously. Existing studies have achieved advanced emotion recognition performance using multimodal knowledge. ...
Automatic inference of cross-modal nonverbal interactions in multiparty conversations: "who responds to whom, when, and how?" from gaze, head gestures, and utterances
ICMI '07: Proceedings of the 9th international conference on Multimodal interfaces

A novel probabilistic framework is proposed for analyzing cross-modal nonverbal interactions in multiparty face-to-face conversations. The goal is to determine "who responds to whom, when, and how" from multimodal cues including gaze, head gestures, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

November 2006

404 pages

ISBN:159593541X

DOI:10.1145/1180995

General Chairs:
Francis Quek
Virginia Tech, USA
,
Jie Yang
Carnegie Mellon University, USA
,
Program Chairs:
Dominic Massaro
University of California, Santa Cruz, USA
,
Abeer Alwan
University of California, Los Angeles, USA
,
Timothy J. Hazen
Massachusetts Institute of Technology, USA

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICMI06

Sponsor:

ICMI06: 8th International Conference on Multimodal Interfaces 2006

November 2 - 4, 2006

Alberta, Banff, Canada

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
281
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yonezawa TYamazoe HUtsumi AAbe S(2008)GazeRoboard: Gaze-communicative guide system in daily life on stuffed-toy robot with interactive display board2008 IEEE/RSJ International Conference on Intelligent Robots and Systems10.1109/IROS.2008.4650692(1204-1209)Online publication date: Sep-2008
https://doi.org/10.1109/IROS.2008.4650692
Yonezawa TYamazoe HUtsumi AAbe SMase KMassaro DTakeda KRoy DPotamianos A(2007)Gaze-communicative behavior of stuffed-toy robot with joint attention and eye contact based on ambient gaze-trackingProceedings of the 9th international conference on Multimodal interfaces10.1145/1322192.1322218(140-145)Online publication date: 12-Nov-2007
https://dl.acm.org/doi/10.1145/1322192.1322218

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten