Skip to main content
SpringerLink
Log in

Computational Multimodal Models of Users’ Interactional Trust in Multiparty Human-Robot Interaction

  • Conference paper
  • First Online:
Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges (ICPR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13643))

Included in the following conference series:

  • 446 Accesses

Abstract

In this paper, we present multimodal computational models of interactional trust in a humans-robot interaction scenario. We address trust modeling as a binary as well as a multi-class classification problem. We also investigate how early- and late-fusion of modalities impact trust modeling. Our results indicate that early-fusion performs better in both the binary and multi-class formulations, meaning that modalities have co-dependencies when studying trust. We also run a SHapley Additive exPlanation (SHAP) values analysis for a Random Forest in the binary classification problem, as it is the model with the best results, to explore which multimodal features are the most relevant to detect trust or mistrust.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aroyo, A.M., et al.: Overtrusting robots: setting a research agenda to mitigate overtrust in automation. Paladyn, J. Beh. Robot. 12(1), 423–436 (2021)

    Article  Google Scholar 

  2. Atamna, A., Clavel, C.: HRI-RNN: a user-robot dynamics-oriented RNN for engagement decrease detection. In: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2020-Oct., pp. 4198–4202 (2020)

    Google Scholar 

  3. Baltrusaitis, T., Zadeh, A., Lim, Y.C., Morency, L.P.: OpenFace 2.0: facial behavior analysis toolkit. In: 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018), pp. 59–66 (2018)

    Google Scholar 

  4. Camurri, A., Mazzarino, B., Volpe, G.: Analysis of expressive gesture: the eyesweb expressive gesture processing library. In: Camurri, A., Volpe, G. (eds.) Gesture-Based Communication in Human-Computer Interaction, pp. 460–467 (2004)

    Google Scholar 

  5. Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7291–7299 (2017)

    Google Scholar 

  6. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)

    Article  MATH  Google Scholar 

  7. Chita-Tegmark, M., Law, T., Rabb, N., Scheutz, M.: Can you trust your trust measure? In: Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction, pp. 92–100 (2021)

    Google Scholar 

  8. Dunn, O.J.: Multiple comparisons using rank sums. Technometrics 6(3), 241–252 (1964)

    Article  Google Scholar 

  9. Duranti, A.: The Anthropology of Intentions: Language in a World of Others. Cambridge University Press, Cambridge (2015)

    Google Scholar 

  10. Eyben, F., et al.: The Geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing. IEEE Trans. Affect. Comput. 7(2), 190–202 (2016)

    Article  Google Scholar 

  11. Eyben, F., Wöllmer, M., Schuller, B.: Opensmile: the Munich versatile and fast open-source audio feature extractor. In: Proceedings of the 18th ACM International Conference on Multimedia, pp. 1459–1462 (2010)

    Google Scholar 

  12. Ford, C., Thompson, S.: Interactional units in conversation: syntactic, intonational and pragmatic resources. In: Interaction and Grammar, p. 134 (1996)

    Google Scholar 

  13. Fulmer, C.A., Gelfand, M.J.: At what level (and in whom) we trust: trust across multiple organizational levels. J. Manag. 38(4), 1167–1230 (2012)

    Google Scholar 

  14. Goodwin, C.: Conversational organization. Interaction between speakers and hearers (1981)

    Google Scholar 

  15. Grossman, R., Friedman, S.B., Kalra, S.: Teamwork processes and emergent states. In: The Wiley Blackwell Handbook of the Psychology of Team Working and Collaborative Processes, pp. 243–269. Wiley, March 2017

    Google Scholar 

  16. Hancock, P.A., Billings, D.R., Schaefer, K.E., Chen, J.Y.C., de Visser, E.J., Parasuraman, R.: A meta-analysis of factors affecting trust in human-robot interaction. Hum. Fact. J. Hum. Fact. Ergon. Soc. 53(5), 517–527 (2011)

    Article  Google Scholar 

  17. Hemamou, L., Felhi, G., Vandenbussche, V., Martin, J.C., Clavel, C.: HireNet: a hierarchical attention model for the automatic analysis of asynchronous video job interviews. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 573–581, July 2019

    Google Scholar 

  18. Heritage, J.C.: International accountability: a conversation analytic perspective. Réseaux. Communication-Technologie-Société 8(1), 23–49 (1990)

    Google Scholar 

  19. Hulcelle, M., Varni, G., Rollet, N., Clavel, C.: Turin: a coding system for trust in human robot interaction. In: 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 1–8 (2021)

    Google Scholar 

  20. Jayagopi, D.B., et al.: The vernissage corpus: a conversational Human-Robot-Interaction dataset. ACM/IEEE International Conference on Human-Robot Interaction, pp. 149–150 (2013)

    Google Scholar 

  21. Khalid, H.M., et al.: Exploring psycho-physiological correlates to trust: implications for human-robot-human interaction. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 60, no. 1, pp. 697–701 (2016)

    Google Scholar 

  22. Khavas, Z.R.: A review on trust in human-robot interaction (2021). https://arxiv.org/abs/2105.10045

  23. Khavas, Z.R., Ahmadzadeh, S.R., Robinette, P.: Modeling trust in human-robot interaction: a survey. In: Wagner, A.R., et al. (eds.) ICSR 2020. LNCS (LNAI), vol. 12483, pp. 529–541. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62056-1_44

    Chapter  Google Scholar 

  24. Kozlowski, S.W., Ilgen, D.R.: Enhancing the effectiveness of work groups and teams. Psychol. Sci. Public Interest 7(3), 77–124 (2006)

    Article  Google Scholar 

  25. Kraus, M., Wagner, N., Minker, W.: Modelling and predicting trust for developing proactive dialogue strategies in mixed-initiative interaction. In: Proceedings of the 2021 International Conference on Multimodal Interaction, pp. 131–140 (2021)

    Google Scholar 

  26. Kruskal, W.H., Wallis, W.A.: Use of ranks in one-criterion variance analysis. J. Am. Stat. Assoc. 47(260), 583–621 (1952)

    Article  MATH  Google Scholar 

  27. Lee, J.J., Knox, B., Baumann, J., Breazeal, C., DeSteno, D.: Computationally modeling interpersonal trust. Front. Psychol. 4 (2013)

    Google Scholar 

  28. Lee, J.D., See, K.A.: Trust in automation: designing for appropriate reliance Human Factors. Hum. Factors 46(1), 50–80 (2004)

    Article  Google Scholar 

  29. Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  30. Marks, M.A., Mathieu, J.E., Zaccaro, S.J.: A temporally based framework and taxonomy of team processes. Acad. Manag. Rev. 26(3), 356–376 (2001)

    Article  Google Scholar 

  31. Mayer, R.C., Davis, J.H., Schoorman, F.D.: An integrative model of organizational trust. Acad. Manag. Rev. 20(3), 709–735 (1995)

    Article  Google Scholar 

  32. Mumm, J., Mutlu, B.: Human-robot proxemics: physical and psychological distancing in human-robot interaction. In: HRI 2011 - Proceedings of the 6th ACM/IEEE International Conference on Human-Robot Interaction, pp. 331–338 (2011)

    Google Scholar 

  33. Oertel, C., et al.: Engagement in human-agent interaction: an overview. Front. Roboti. AI 7 (2020)

    Google Scholar 

  34. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  35. Rapp, T., Maynard, T., Domingo, M., Klock, E.: Team emergent states: What has emerged in the literature over 20 years. Small Group Res. 52, 68–102 (2021)

    Article  Google Scholar 

  36. Rousseau, D.M., Sitkin, S.B., Burt, R.S., Camerer, C., Rousseau, D.M., Burt, R.S.: Not So Different After All : a Cross-Discipline View of Trust. Acad. Manag. Rev. 23(3), 393–404 (1998)

    Article  Google Scholar 

  37. Schaefer, K.E.: The Perception and Measurement of Human-robot Trust. Doctoral Dissertation. University of Central Florida, Orlando (2013)

    Google Scholar 

  38. Stivers, T.: Stance, alignment, and affiliation during storytelling: when nodding is a token of affiliation. Res. Lang. Soc. Interact. 41(1), 31–57 (2008)

    Article  Google Scholar 

  39. Syrdal, D.S., Dautenhahn, K., Koay, K.L., Walters, M.L.: The negative attitudes towards robots scale and reactions to robot behaviour in a live human-robot interaction study. In: Adaptive and emergent behaviour and complex systems (2009)

    Google Scholar 

  40. Tutul, A.A., Nirjhar, E.H., Chaspari, T.: Investigating trust in human-machine learning collaboration: a pilot study on estimating public anxiety from speech. In: Proceedings of the 2021 International Conference on Multimodal Interaction, pp. 288–296 (2021)

    Google Scholar 

  41. Wilcoxon, F.: Individual comparisons by ranking methods. Biomet. Bull. 1(6), 80–83 (1945)

    Article  Google Scholar 

Download references

Acknowledgment

This work is supported by the Data Science and Artificial Intelligence for Digitalized Industry and Services (DSAIDIS) chair of Télécom Paris, and the European project H2020 ANIMATAS (ITN 7659552).

Author information

Authors and Affiliations

  1. LTCI Laboratory, Télécom-Paris, Institut Polytechnique de Paris, 91123, Palaiseau, France

    Marc Hulcelle, Giovanna Varni & Chloé Clavel

  2. ’Institut Interdisciplinaire de l’Innovation’ Laboratory, Télécom-Paris, Institut Polytechnique de Paris, 91123, Palaiseau, France

    Nicolas Rollet

Authors
  1. Marc Hulcelle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giovanna Varni
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicolas Rollet
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chloé Clavel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Marc Hulcelle , Giovanna Varni or Chloé Clavel .

Editor information

Editors and Affiliations

  1. York University, Toronto, ON, Canada

    Jean-Jacques Rousseau

  2. Ontario Tech University, Oshawa, ON, Canada

    Bill Kapralos

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hulcelle, M., Varni, G., Rollet, N., Clavel, C. (2023). Computational Multimodal Models of Users’ Interactional Trust in Multiparty Human-Robot Interaction. In: Rousseau, JJ., Kapralos, B. (eds) Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges. ICPR 2022. Lecture Notes in Computer Science, vol 13643. Springer, Cham. https://doi.org/10.1007/978-3-031-37660-3_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-37660-3_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-37659-7

  • Online ISBN: 978-3-031-37660-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation