Abstract
In face-to-face conversations, speakers monitor the listener’s gaze to check whether the listener is engaged in the conversation. The speaker may change the conversational strategy if the listener is not fully engaged in the conversation. In this chapter, we propose an algorithm to estimate the user’s conversational engagement based on various types of gaze information, such as gaze shift patterns, gaze duration, amount of eye movement, and pupil size. By applying the proposed algorithm, we implement an agent that can change its conversational strategy according to the user’s conversational engagement. We also evaluate the agent system by investigating how the agent’s awareness of the user’s engagement affects the user’s verbal and nonverbal behaviors as well as the subjective impressions of the agent. First, based on an empirical study, we identify useful information for estimating user engagement, and establish an engagement estimation model using a decision tree technique. The model can predict the user’s disengagement with an accuracy of over 70 %. Then, the model is implemented as a real-time engagement-judgment mechanism and is incorporated into a multimodal dialogue manager in a conversational agent. Finally, our evaluation experiment reveals that probing questions by the engagement-sensitive agent successfully recover the subject’s conversational engagement, change the gaze behaviors of the subject, and elicit more verbal contribution. Moreover, such timely probing questions also improve the subject’s impression of the agent.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
julius-4.0.2. Available from http://julius.sourceforge.jp/forum/viewtopic.php?f=13&t=53.
References
Anderson AH et al. (1997) The effects of face-to-face communication on the intelligibility of speech. Percept Psychophys 59:580–592
Argyle M, Cook M (1976) Gaze and mutual gaze. Cambridge University Press, Cambridge
Argyle M, Graham J (1977) The Central Europe experiment—looking at persons and looking at things. J Environ Psychol Nonverbal Behav 1:6–16
Argyle M et al. (1973) The different functions of gaze. Semiotica 7:19–32
Bohus D, Horvitz E (2009) Learning to predict engagement with a spoken dialog system in open-world settings. In: SIGdial’09, London, pp 244–252
Clark HH (1996) Using language. Cambridge University Press, Cambridge
Eichner T et al (2007) Attentive presentation agents. In: The 7th international conference on intelligent virtual agents (IVA), pp 283–295
Hess EH (1965) Attitude and pupil size. Sci Am 212:46–54
Iqbal ST, Bailey BP (2004) Task-evoked pupillary response to mental workload in human-computer interaction. In: CHI’04, Vienna, pp 1477–1480
Iqbal ST, Xianjun SZ, Bailey BP (2005) Towards an index of opportunity: understanding changes in mental workload during task execution. In: CHI’05, Portland
Kendon A (1967) Some functions of gaze direction in social interaction. Acta Psychol 26:22–63
Kipp M (2001) Anvil—a generic annotation tool for multimodal dialogue. In: The 7th European conference on speech communication and technology, pp 1367–1370
Matheson C, Poesio M, Traum D (2000) Modelling grounding and discourse obligations using update rules. In: 1st annual meeting of the North American chapter of the Association for Computational Linguistics (NAACL2000), pp 1–8
Morency L-P et al. (2007) Head gestures for perceptual interfaces: the role of context in improving recognition. Artif Intell 171(8–9):568–585
Nakano YI, Nishida T (2007) Attentional behaviors as nonverbal communicative signals in situated interactions with conversational agents. In: Nishida T (ed) Engineering approaches to conversational informatics. Wiley, New York, pp 85–102
Nakano YI et al (2003) Towards a model of face-to-face grounding. In: The 41st annual meeting of the Association for Computational Linguistics (ACL03), Sapporo, Japan, pp 553–561
Nakano YI et al (2004) Converting text into agent animations: assigning gestures to text. In: Human language technology conference of the North American chapter of the Association for Computational Linguistics (HLT-NAACL 2004), Boston, companion volume, pp 91–102
Novick DG, Hansen B, Ward K (1996) Coordinating turn-taking with gaze. In: ICSLP-96, Philadelphia, vol 3, pp 1888–1891
Qvarfordt P, Zhai S (2005) Conversing with the user based on eye-gaze patterns. In: The conference on human-factors in computing systems (CHI 2005), pp 221–230
Remco R, Bouckaert EF, Hall MA, Holmes G, Pfahringer B, Reutemann P, Witten IH (2010) WEKA—experiences with a java open-source project. J Mach Learn Res 11:2533–2541
Rich C et al (2010) Recognizing engagement in human-robot interaction. In: ACM/IEEE international conference on human-robot interaction, pp 375–382
Sidner CL et al (2004) Where to look: a study of human-robot engagement. In: ACM international conference on intelligent user interfaces (IUI), pp 78–84
Whittaker S (2003) Theories and methods in mediated communication. In: Graesser A, Gernsbacher M, Goldman S (eds) The handbook of discourse processes. Erlbaum, Hillsdale, pp 243–286
Acknowledgement
This study was funded in part by JSPS under a Grant-in-Aid for Scientific Research (S) (19100001).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag London
About this chapter
Cite this chapter
Ishii, R., Ooko, R., Nakano, Y.I., Nishida, T. (2013). Effectiveness of Gaze-Based Engagement Estimation in Conversational Agents. In: Nakano, Y., Conati, C., Bader, T. (eds) Eye Gaze in Intelligent User Interfaces. Springer, London. https://doi.org/10.1007/978-1-4471-4784-8_6
Download citation
DOI: https://doi.org/10.1007/978-1-4471-4784-8_6
Publisher Name: Springer, London
Print ISBN: 978-1-4471-4783-1
Online ISBN: 978-1-4471-4784-8
eBook Packages: Computer ScienceComputer Science (R0)