Abstract
In complex conversation tasks, people react to their interlocutor’s state, such as uncertainty and engagement to improve conversation effectiveness Forbes-Riley and Litman (Adapting to student uncertainty improves tutoring dialogues, pp 33–40, 2009 [2]). If a conversational system reacts to a user’s state, would that lead to a better conversation experience? To test this hypothesis, we designed and implemented a dialog system that tracks and reacts to a user’s state, such as engagement, in real time. We designed and implemented a conversational job interview task based on the proposed framework. The system acts as an interviewer and reacts to user’s disengagement in real-time with positive feedback strategies designed to re-engage the user in the job interview process. Experiments suggest that users speak more while interacting with the engagement-coordinated version of the system as compared to a non-coordinated version. Users also reported the former system as being more engaging and providing a better user experience.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Baltru T, Robinson P, Morency L-P et al (2016) Openface: an open source facial behavior analysis toolkit. In: 2016 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 1–10
Forbes-Riley K, Litman DJ (2009) Adapting to student uncertainty improves tutoring dialogues. In: AIED, pp 33–40
He X, Yan S, Hu Y, Niyogi P, Zhang H-J (2005) Face recognition using laplacian faces. IEEE Trans Pattern Anal Mach Intell 27(3):328–340
Kousidis S, Kennington C, Baumann T, Buschmeier H, Kopp S, Schlangen D (2014) A multimodal in-car dialogue system that tracks the driver’s attention. In: Proceedings of the 16th international conference on multimodal interaction. ACM, pp 26–33
Lamere P, Kwok P, Gouvea E, Raj B, Singh R, Walker W, Warmuth M, Wolf P (2003) The cmu sphinx-4 speech recognition system. In: IEEE international conference on acoustics, speech and signal processing (ICASSP 2003), Hong Kong, vol 1, pp 2–5 (Citeseer)
Lehman B, DMello S, Graesser A (2012) Interventions to regulate confusion during learning. In: International conference on intelligent tutoring systems. Springer, pp 576–578
Morency L-P, Sidner C, Lee C, Darrell T (2005) Contextual recognition of head gestures. In: Proceedings of the 7th international conference on multimodal interfaces. ACM, pp 18–24
Ramanarayanan V, Suendermann-Oeft D, Lange P, Mundkowsky R, Ivanou A, Yu Z, Qian Y, Evanini K (2016) Assembling the jigsaw: how multiple w3c standards are synergistically combined in the halef multimodal dialog system. In: Multimodal interaction with W3C standards: towards natural user interfaces to everything. Springer, page to appear
Schnelle-Walka D, Radomski S, Mühlhäuser M (2013) Jvoicexml as a modality component in the w3c multimodal architecture. J Multimodal User Interfaces 7(3):183–194
Schröder M, Trouvain J (2003) The german text-to-speech synthesis system mary: a tool for research, development and teaching. Int J Speech Technol 6(4):365–377
Sciutti A, Schillingmann L, Palinko O, Nagai Y, Sandini G (2015) A gaze-contingent dictating robot to study turn-taking. In: Proceedings of the tenth annual acm/ieee international conference on human-robot interaction extended abstracts. ACM, pp 137–138
Taylor P, Black AW, Caley R (1998) The architecture of the festival speech synthesis system
Van Meggelen J, Madsen L, Smith J (2007) Asterisk: the future of telephony. O’Reilly Media, Inc
Vinciarelli A, Pantic M, Bourlard H (2009) Social signal processing: survey of an emerging domain. Image Vis Comput J 27(12):1743–1759
Wendler D (2014) Improve your social skills. CreateSpace Independent Publishing Platform
Whitehill J, Serpell Z, Lin Y-C, Foster A, Movellan JR (2014) The faces of engagement: automatic recognition of student engagementfrom facial expressions. IEEE Trans Affect Comput 5(1):86–98
Yu Z, Bohus D, Horvitz E (2015) Incremental coordination: attention-centric speech production in a physically situated conversational agent. In: 16th annual meeting of the special interest group on discourse and dialogue, p 402
Yu Z, Gerritsen D, Ogan A, Black AW, Cassell J (2013) Automatic prediction of friendship via multi-model dyadic features. In: Proceedings of SIGDIAL, pp 51–60
Yu Z, Nicolich-Henkin L, Black A, Rudnicky A (2016) A wizard-of-oz study on a non-task-oriented dialog systems that reacts to user engagement. In: 17th annual meeting of the special interest group on discourse and dialogue
Yu Z, Ramanarayanan V, Mundkowsky R, Lange P, Ivanov A, Black AW, Suendermann-Oeft D (2016) Multimodal halef: an open-source modular web-based multimodal dialog framework
Acknowledgements
We would like to thank Robert Mundkowsky and Dmytro Galochkin for help with system engineering. We would also like to thank Eugene Tsuprun, Keelan Evanini, Nehal Sadek and Liz Bredlau for help with the task design and useful discussions.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer International Publishing AG, part of Springer Nature
About this chapter
Cite this chapter
Yu, Z., Ramanarayanan, V., Lange, P., Suendermann-Oeft, D. (2019). An Open-Source Dialog System with Real-Time Engagement Tracking for Job Interview Training Applications. In: Eskenazi, M., Devillers, L., Mariani, J. (eds) Advanced Social Interaction with Agents . Lecture Notes in Electrical Engineering, vol 510. Springer, Cham. https://doi.org/10.1007/978-3-319-92108-2_21
Download citation
DOI: https://doi.org/10.1007/978-3-319-92108-2_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-92107-5
Online ISBN: 978-3-319-92108-2
eBook Packages: EngineeringEngineering (R0)