skip to main content
10.1145/3281151.3281153acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

Estimating interviewee's willingness in multimodal human robot interview interaction

Published: 16 October 2018 Publication History

Abstract

This study presents a prediction model of a speaker's willingness level in human-robot interview interaction by using multimodal features (i.e., verbal, audio, and visual). We collected a novel multimodal interaction corpus, including two types of annotation data sets of willingness. A binary classification task of the willingness level (high or low) was implemented to evaluate the proposed multimodal prediction model. We obtained the best classification accuracy (i.e., 0.6) using the random forest model with audio and motion features. The difference between best accuracy (i.e., 0.6) and coder's recognition accuracy (i.e., 0.73) was 0.13.

References

[1]
2004. Speech Prosody Analysis Software Tools. (2004). http://affect.media.mit.edu/downloads/SpeechProsodyAnalysisTools.tar.
[2]
Masahiro Araki, Sayaka Tomimasu, Mikio Nakano, Kazunori Komatani, Shogo Okada, Shinya Fujie, and Hiroaki Sugiyama. 2018. Collection of Multimodal Dialog Data and Analysis of the Result of Annotation of Users' Interest Level. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7--12, 2018.
[3]
Leo Breiman. 2001. Random Forests. Machine Learning 45, 1 (2001), 5--32.
[4]
Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). ACM, New York, NY, USA, 785--794.
[5]
Yuya Chiba, Masashi Ito, Takashi Nose, and Akinori Ito. 2014. User Modeling by Using Bag-of-Behaviors for Building a Dialog System Sensitive to the Interlocutor's Internal State. In Proc. Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL). 74--78. http://www.aclweb.org/anthology/W14-4310.
[6]
Nadine Glas and Catherine Pelachaud. 2015. Definitions of engagement in human-agent interaction. In International Workshop on Engagment in Human Computer Interaction (ENHANCE). 944--949.
[7]
Takatsugu Hirayama, Yasuyuki Sumi, Tatsuya Kawahara, and Takashi Matsuyama. 2011. Info-concierge: Proactive multi-modal interaction through mind probing. In The Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC 2011).
[8]
Mohammed Ehsan Hoque, Matthieu Courgeon, Jean-Claude Martin, Bilge Mutlu, and Rosalind W Picard. 2013. Mach: My automated conversation coach. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 697--706.
[9]
Koji Inoue, Divesh Lala, Katsuya Takanashi, and Tatsuya Kawahara. 2018. Latent character model for engagement recognition based on multimodal behaviors. In Proc. Int'l Workshop Spoken Dialogue Systems (IWSDS).
[10]
Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying Conditional Random Fields to Japanese Morphological Analysis. In Proceedings on Empirical Methods in Natural Language Processing (EMNLP), Vol. 4. 230--237. http://taku910.github.io/mecab
[11]
Akinobu Lee and Tatsuya Kawahara. 2009. Recent Development of Open-Source Speech Recognition Engine Julius. In Proceedings Asia-Pacific Signal and Information Processing Association, Annual Summit and Conference (APSIPA ASC:). 131--137.
[12]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.
[13]
Hanan Salam, Oya Celiktutan, Isabelle Hupont, Hatice Gunes, and Mohamed Chetouani. 2017. Fully automatic analysis of engagement and its relationship to personality in human-robot interactions. IEEE Access 5 (2017), 705--721.
[14]
Giota Stratou and Louis-Philippe Morency. 2017. MultiSense---Context-Aware Nonverbal Behavior Analysis Framework: A Psychological Distress Use Case. IEEE Transactions on Affective Computing 8, 2 (2017), 190--203.
[15]
Hiroki Tanaka, Hiroyoshi Adachi, Norimichi Ukita, Manabu Ikeda, Hiroaki Kazui, Takashi Kudo, and Satoshi Nakamura. 2017a. Detecting Dementia Through Interactive Computer Avatars. IEEE journal of translational engineering in health and medicine 5 (2017), 1--11.
[16]
Hiroki Tanaka, Hideki Negoro, Hidemi Iwasaka, and Satoshi Nakamura. 2017b. Embodied conversational agents for multimodal automated social skills training in people with autism spectrum disorders. PloS one 12, 8 (2017), e0182151.
[17]
Alessandro Vinciarelli, Maja Pantic, and Hervé Bourlard. 2009. Social signal processing: Survey of an emerging domain. Image and vision computing 27, 12 (2009), 1743--1759.

Cited By

View all
  • (2024)Adaptive Interview Strategy Based on Interviewees’ Speaking Willingness Recognition for Interview RobotsIEEE Transactions on Affective Computing10.1109/TAFFC.2023.330964015:3(942-957)Online publication date: Jul-2024
  • (2021)Dialogue Management by Estimating User’s Internal State Using the Movie Recommendation Dialogue映画推薦対話を具体例とした話者内部状態の推定による対話管理Journal of Natural Language Processing10.5715/jnlp.28.10428:1(104-135)Online publication date: 2021
  • (2020)A Job Interview Dialogue System That Asks Follow-up Questions: Implementation and Evaluation with an Autonomous Android掘り下げ質問を行う就職面接対話システムの自律型アンドロイドでの実装と評価Transactions of the Japanese Society for Artificial Intelligence10.1527/tjsai.35-5_D-K4335:5(D-K43_1-10)Online publication date: 1-Sep-2020

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '18: Proceedings of the 20th International Conference on Multimodal Interaction: Adjunct
October 2018
62 pages
ISBN:9781450360029
DOI:10.1145/3281151
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. interview interaction
  2. multimodal machine learning
  3. speaker's willingness

Qualifiers

  • Research-article

Conference

ICMI '18
Sponsor:

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)10
  • Downloads (Last 6 weeks)1
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Adaptive Interview Strategy Based on Interviewees’ Speaking Willingness Recognition for Interview RobotsIEEE Transactions on Affective Computing10.1109/TAFFC.2023.330964015:3(942-957)Online publication date: Jul-2024
  • (2021)Dialogue Management by Estimating User’s Internal State Using the Movie Recommendation Dialogue映画推薦対話を具体例とした話者内部状態の推定による対話管理Journal of Natural Language Processing10.5715/jnlp.28.10428:1(104-135)Online publication date: 2021
  • (2020)A Job Interview Dialogue System That Asks Follow-up Questions: Implementation and Evaluation with an Autonomous Android掘り下げ質問を行う就職面接対話システムの自律型アンドロイドでの実装と評価Transactions of the Japanese Society for Artificial Intelligence10.1527/tjsai.35-5_D-K4335:5(D-K43_1-10)Online publication date: 1-Sep-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media