Skip to main content
SpringerLink
Log in

Investigation on the Fusion of Multi-modal and Multi-person Features in RNNs for Detecting the Functional Roles of Group Discussion Participants

  • Conference paper
  • First Online:
Social Computing and Social Media. Design, Ethics, User Behavior, and Social Network Analysis (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12194))

Included in the following conference series:

  • 4533 Accesses

Abstract

More and more companies are putting emphasis on communication skill in the recruitment of their employees and adopt group discussion as part of their recruitment interview. In our ongoing project, we aim to develop a training system that can provide advices to its users in improving the perception of their communication skill during group discussion. In order to realize this goal, a conceptual unit of communicational behaviors and a template of communication style are required. We propose the use of functional roles of the participants in group discussions as this unit. In order to incorporate the use of functional roles for improving the perception of participants’ communication skill, the first task is automatic detection of the participants’ functional roles in real-time. We previously proposed a SVM based model for this task but the results were only moderate. We expect including temporal characteristics, frame-wise interaction of modalities, and inter-person interaction can improve the classification accuracy and explored the use of RNN based networks to see the effectiveness of these factors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.fon.hum.uva.nl/praat/.

  2. 2.

    https://github.com/TadasBaltrusaitis/OpenFace.

  3. 3.

    https://github.com/CMU-Perceptual-Computing-Lab/openpose.

  4. 4.

    https://www.audeering.com/what-we-do/opensmile/.

References

  1. Aran, O., Gatica-Perez, D.: One of a kind: inferring personality impressions in meetings. In: Proceedings of 15th ACM International Conference on Multimodal Interaction (ICMI 2013), Sydney, Australia, December 2013

    Google Scholar 

  2. Argyle, M., Cook, M.: Gaze and Mutual Gaze. Cambridge University Press, Cambridge (1976)

    Google Scholar 

  3. Bengio, Y., Glorot, X., Bordes, A.: Deep sparse rectifier neural networks. In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), vol. 15, pp. 315–323 (2011)

    Google Scholar 

  4. Benne, K.D., Sheats, P.: Functional roles of group members. J. Soc. Issues 4(2), 41–49 (1948)

    Article  Google Scholar 

  5. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)

    Article  Google Scholar 

  6. Clark, H.H., Carlson, T.B.: Hearers and speech acts. Language 58(2), 332–373 (1982)

    Article  Google Scholar 

  7. Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System (FACS). Website (2002). http://www.face-and-emotion.com/dataface/facs/description.jsp

  8. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

    MATH  Google Scholar 

  9. Hall, J.A., Coats, E.J., LeBeau, L.S.: Nonverbal behavior and the vertical dimension of social relations: a meta-analysis. Psychol. Bull. 131(6), 898–924 (2005)

    Article  Google Scholar 

  10. Hare, P.: Types of roles in small groups, a bit history and a current perspective. Small Group Res. 25(3), 433–448 (1994)

    Article  Google Scholar 

  11. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  12. Huang, H.H., Zhang, Q., Okada, S., Kuwabara, K., Nishida, T.: Adopting functional roles for improving participants’ communication skill in group discussion conversation. In: Workshop on Group Interaction Frontiers in Technology (GIFT 2018), 20th ACM International Conference on Multimodal Interaction (ICMI 2018), Boulder, USA, October 2018

    Google Scholar 

  13. Institute, H.R.: Report of the 2017 investigation on the trend in the recruitment of new graduates (2017). (in Japanese)

    Google Scholar 

  14. Keidanren Japan Business Federation: Reports of the 2017 investigation on the recruitment of new graduates, November 2017. (in Japanese)

    Google Scholar 

  15. Kendon, A.: Some functions of gaze direction in social interaction. Acta Psychol. 26, 22–63 (1967)

    Article  Google Scholar 

  16. Lausberg, H., Sloetjes, H.: Coding gestural behavior with the NEUROGES-ELAN system. Behav. Res. Methods 41(3), 841–849 (2009)

    Article  Google Scholar 

  17. McNeill, D.: Hand and Mind. The University of Chicago Press, Chicago (1992)

    Google Scholar 

  18. Muralidhar, S., Nguyen, L.S., Frauendorfer, D., Odobez, J.M., Mast, M.S., Gatica-Perez, D.: Training on the job: behavioral analysis of job interviews in hospitality. In: 18th ACM International Conference on Multimodal Interaction (ICMI 2016), Tokyo, Japan, pp. 84–91, November 2016

    Google Scholar 

  19. Nihei, F., Nakano, Y.I., Hayashi, Y., Huang, H.H., Okada, S.: Predicting influential statements in group discussions using speech and head motion information. In: 16th International Conference on Multimodal Interaction (ICMI 2014), Istanbul, pp. 136–143, November 2014

    Google Scholar 

  20. Okada, S., Nakano, Y., Hayashi, Y., Takase, Y., Nitta, K.: Estimating communication skills using dialogue acts and nonverbal features in multiple discussion datasets. In: 18th ACM International Conference on Multimodal Interaction (ICMI 2016), Tokyo, pp. 169–176, November 2016

    Google Scholar 

  21. Raducanu, B., Vitria, J., Gatica-Perez, D.: You are fired! Nonverbal role analysis in competitive meetings. In: 2009 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2009), Taipei, Taiwan, April 2009

    Google Scholar 

  22. Schiavo, G., Cappelletti, A., Mencarini, E., Stock, O., Zancanaro, M.: Overt or subtle? Supporting group conversations with automatically targeted directives. In: Proceedings of the 19th International Conference on Intelligent User Interfaces (IUI 2014), pp. 225–234 (2014)

    Google Scholar 

  23. Schuller, B., Steidl, S., Batliner, A.: The interspeech 2009 emotion challenge. In: 10th Annual Conference of the International Speech Communication Association (Interspeech 2009), Brighton, United Kingdom, September 2009

    Google Scholar 

  24. Zancanaro, M., Lepri, B., Pianesi, F.: Automatic detection of group functional roles in face to face interactions. In: Proceedings of the 8th International Conference on Multimodal Interfaces (ICMI 2006), pp. 28–34 (2006)

    Google Scholar 

Download references

Acknowledgements

This research is partially supported by KAKENHI: Grant-in-Aid for Scientific Research (A), Grant No. 19H01120.

Author information

Authors and Affiliations

  1. Faculty of Informatics, University of Fukuchiyama, Fukuchiyama, Japan

    Hung-Hsuan Huang & Toyoaki Nishida

  2. Center for Advanced Intelligence Project, RIKEN, Kyoto, Japan

    Hung-Hsuan Huang & Toyoaki Nishida

Authors
  1. Hung-Hsuan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Toyoaki Nishida
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hung-Hsuan Huang .

Editor information

Editors and Affiliations

  1. Towson University, Towson, MD, USA

    Gabriele Meiselwitz

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Huang, HH., Nishida, T. (2020). Investigation on the Fusion of Multi-modal and Multi-person Features in RNNs for Detecting the Functional Roles of Group Discussion Participants. In: Meiselwitz, G. (eds) Social Computing and Social Media. Design, Ethics, User Behavior, and Social Network Analysis. HCII 2020. Lecture Notes in Computer Science(), vol 12194. Springer, Cham. https://doi.org/10.1007/978-3-030-49570-1_34

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-49570-1_34

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-49569-5

  • Online ISBN: 978-3-030-49570-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation