skip to main content
10.1145/3136755.3143034acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
short-paper

A modular, multimodal open-source virtual interviewer dialog agent

Published: 03 November 2017 Publication History

Abstract

We present an open-source multimodal dialog system equipped with a virtual human avatar interlocutor. The agent, rigged in Blender and developed in Unity with WebGL support, interfaces with the HALEF open-source cloud-based standard-compliant dialog framework. To demonstrate the capabilities of the system, we designed and implemented a conversational job interview scenario where the avatar plays the role of an interviewer and responds to user input in real-time to provide an immersive user experience.

References

[1]
Sandra Baldassarri, Eva Cerezo, and Francisco J Seron. 2008. Maxine: A platform for embodied animated agents. Computers & Graphics 32, 4 (2008), 430–437.
[2]
Pascale Fung, Dario Bertero, Yan Wan, Anik Dey, Ricky Ho Yin Chan, Farhad Bin Siddique, Yang Yang, Chien-Sheng Wu, and Ruixi Lin. 2016. Towards Empathetic Human-Robot Interactions. arXiv preprint arXiv:1605.04072 (2016).
[3]
Mohammed Ehsan Hoque, Matthieu Courgeon, Jean-Claude Martin, Bilge Mutlu, and Rosalind W Picard. 2013. Mach: My automated conversation coach. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 697–706.
[4]
SKSEA James and Ian Vales. 2009. Personal healthcare assistant/companion in virtual world. In AAAI Fall Symposium Series.
[5]
Shin-ichi Kawamoto, Hiroshi Shimodaira, Tsuneo Nitta, Takuya Nishimoto, Satoshi Nakamura, Katsunobu Itou, Shigeo Morishima, Tatsuo Yotsukura, Atsuhiko Kai, Akinobu Lee, et al. 2004. Galatea: Open-source software for developing anthropomorphic spoken dialog agents. In Life-Like Characters. Springer, 187–211.
[6]
Sungjin Lee, Hyungjong Noh, Jonghoon Lee, Kyusong Lee, and Gary Geunbae Lee. 2010. Cognitive effects of robot-assisted language learning on oral skills. In INTERSPEECH 2010 Satellite Workshop on Second Language Studies: Acquisition, Learning, Education and Technology.
[7]
Vikram Ramanarayanan, David Suendermann-Oeft, Alexei Ivanov, and Keelan Evanini. 2015. A distributed cloud-based dialog system for conversational application development. In 16th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL 2015), Prague, Czech Republic.
[8]
Vikram Ramanarayanan, David Suendermann-Oeft, Patrick Lange, Robert Mundkowsky, Aliaksei Ivanou, Zhou Yu, Yao Qian, and Keelan Evanini. 2016. Assembling the jigsaw: How multiple W3C standards are synergistically combined in the HALEF multimodal dialog system. In Multimodal Interaction with W3C Standards: Towards Natural User Interfaces to Everything. Springer, to appear.
[9]
Thomas Rist, Elisabeth André, Stephan Baldes, Patrick Gebhard, Martin Klesen, Michael Kipp, Peter Rist, and Markus Schmitt. 2004. A review of the development of embodied presentation agents and their application fields. In Life-Like Characters. Springer, 377–404.
[10]
Marcus Thiebaux, Stacy Marsella, Andrew N Marshall, and Marcelo Kallmann. 2008. Smartbody: Behavior realization for embodied conversational agents. In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 151–158. Abstract 1 Introduction 2 System Design and Implementation 2.1 The HALEF Dialog System 2.2 Building the Avatar 2.3 Interfacing the Avatar with HALEF 3 Conclusions 4 Acknowledgments References

Cited By

View all
  • (2023)"This machine is for the aides": Tailoring Voice Assistant Design to Home Health Care WorkProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581118(1-19)Online publication date: 19-Apr-2023
  • (2018)User Affect and No-Match Dialogue ScenariosProceedings of the 4th International Workshop on Multimodal Analyses Enabling Artificial Agents in Human-Machine Interaction10.1145/3279972.3279979(6-14)Online publication date: 16-Oct-2018

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction
November 2017
676 pages
ISBN:9781450355438
DOI:10.1145/3136755
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. avatar
  2. dialog system
  3. multimodal
  4. open-source
  5. virtual agent

Qualifiers

  • Short-paper

Conference

ICMI '17
Sponsor:

Acceptance Rates

ICMI '17 Paper Acceptance Rate 65 of 149 submissions, 44%;
Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1
  • Downloads (Last 6 weeks)0
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)"This machine is for the aides": Tailoring Voice Assistant Design to Home Health Care WorkProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581118(1-19)Online publication date: 19-Apr-2023
  • (2018)User Affect and No-Match Dialogue ScenariosProceedings of the 4th International Workshop on Multimodal Analyses Enabling Artificial Agents in Human-Machine Interaction10.1145/3279972.3279979(6-14)Online publication date: 16-Oct-2018

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media