Skip to main content
Springer Nature Link
Log in

An Overview of Emotion Recognition from Body Movement

  • Conference paper
  • First Online:
Complex, Intelligent and Software Intensive Systems (CISIS 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 497))

  • 719 Accesses

Abstract

Understanding human emotions has become a popular research and practical topic in recent years. There are many research works that have good results on detecting human emotions from facial expressions, speech and text. Recognizing emotions from body posture or movement is an emerging area of research, and it has shown progressive results. This brief survey presents an overview of recent research in this field. The relationship between emotions and body movements is discussed. The factors that affect this relation are presented. Based on recent advanced research, an integrated and comprehensive process for the automatic detection of emotions based on body movements is introduced. Each component of this process is considered. In particular, body movement models, their evaluation, and available datasets are examined.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Schindler, K., Van Gool, L., De Gelder, B.: Recognizing emotions expressed by body pose: a biologically inspired neural model. Neural Netw. 21, 1238–1246 (2008)

    Article  Google Scholar 

  2. Elman, J.L.: Encyclopedia of Language and Linguistics, 2nd edn. Elsevier, Oxford (2005)

    Google Scholar 

  3. De Silva, P., Bianchi-Berthouze, N.: Modeling human affective postures: an information theoretic characterization of posture features. Comput. Anim. Virtual Worlds 15, 269–276 (2004)

    Article  Google Scholar 

  4. Thanh Nguyen, D., Li, W., Ogunbona, P.: Human detection from images and videos: a survey. Pattern Recogn. 51, 148–175 (2016)

    Article  Google Scholar 

  5. Kachouane, M., Sahki, S., Lakrouf, M., Ouadah, N.: Hog based fast human detection. In: 24th International Conference on Microelectronics (ICM), pp. 1–4 (2012)

    Google Scholar 

  6. Dalal, N., Triggs, B., Schmid, C.: Human detection using oriented histograms of flow and appearance. In: European Conference on Computer Vision, pp. 428–441 (2006)

    Google Scholar 

  7. Noroozi, F., Adrian Corneanu, C., Kamińska, D., Sapiński, T., Escalera, S., Anbarjafari, G.: Survey on emotional body gesture recognition. IEEE Trans. Affect. Comput. 12, 505–523 (2018)

    Article  Google Scholar 

  8. Ansari, M., Kumar Singh, D.: Human detection techniques for real time surveillance: a comprehensive survey. Multimedia Tools Appl. 80, 8759–8808 (2021)

    Article  Google Scholar 

  9. Chen, Y., Tian, Y., He, M.: Monocular human pose estimation: a survey of deep learning-based methods. Comput. Vis. Image Underst. 192, 102897 (2020)

    Article  Google Scholar 

  10. Toshev, A., Szegedy, C.: Deeppose: human pose estimation via deep neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1653–1660 (2014)

    Google Scholar 

  11. Charles, J., Pfister, T., Magee, D., Hogg, D., Zisserman, A.: Personalizing human video pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3063–3072 (2016)

    Google Scholar 

  12. Pishchulin, L., et al.: Deepcut: joint subset partition and labeling for multi person pose estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4929–4937 (2016)

    Google Scholar 

  13. Alldieck, T., Magnor, M., Xu, W., Theobalt, C., Pons-Moll, G.: Video based reconstruction of 3d people models. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 8387–8397 (2018)

    Google Scholar 

  14. Crane, E., Gross, M.: Motion capture and emotion: affect detection in whole body movement. In: International Conference on Affective Computing and Intelligent Interaction, pp. 95–101 (2007)

    Google Scholar 

  15. Gunes, H., Piccardi, M.: Affect recognition from face and body: early fusion vs. late fusion. In: IEEE International Conference on Systems, Man and Cybernetics, vol. 4, pp. 3437–3443 (2005)

    Google Scholar 

  16. Maret, Y., Oberson, D., Gavrilova, M.: Identifying an emotional state from body movements using genetic-based algorithms. In: International Conference on Artificial Intelligence and Soft Computing, pp. 474–485 (2018)

    Google Scholar 

  17. Randhavane, T., Bhattacharya, U., Kapsaskis, K., Gray, K., Bera, A., Manocha, D.: Identifying emotions from walking using affective and deep features. arXiv preprintarXiv:1906.11884 (2019)

  18. Ferdous, A., Hossain, B.A.S.M., Marina, G.: Emotion recognition from body movement. IEEE Access 8, 11761–11781 (2019)

    Google Scholar 

  19. Karg, M., Kühnlenz, K., Buss, M.: Recognition of affect based on gait patterns. IEEE Trans. Syst. Man Cybernet. Part B (Cybernet.) 40, 1050–1061 (2010)

    Article  Google Scholar 

  20. Kapur, A., Kapur, A., Virji-Babul, N., Tzanetakis, G., Driessen, P.: Gesture-based affective computing on motion capture data. In: International Conference on Affective Computing and Intelligent Interaction, pp. 1–7 (2005)

    Google Scholar 

  21. Sapiński, T., Kamińska, D., Pelikant, A., Anbarjafari, G.: Emotion recognition from skeletal movement. Entropy 21, 646 (2019)

    Article  Google Scholar 

  22. Kaza, K., et al.: Body motion analysis for emotion recognition in serious games. In: International Conference on Universal Access in Human-Computer Interaction, pp. 33–42 (2016)

    Google Scholar 

  23. Coulson, M.: Attributing emotion to static body postures: recognition accuracy, confusions, and viewpoint dependence. Nonverbal Behav. 28, 117–139 (2004)

    Article  Google Scholar 

  24. Posner, J., Russell, J., Peterson, B.: The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17, 715–734 (2005)

    Article  Google Scholar 

  25. Bernhardt, D.: Emotion inference from human body motion. Technical report, University of Cambridge, Computer Laboratory (2010)

    Google Scholar 

  26. Viegas, C.: Two stage emotion recognition using frame-level and video-level features. In: 15th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 912–915 (2020)

    Google Scholar 

  27. Alghowinem, S., Goecke, R., Cohn, J., Wagner, M., Parker, G., Breakspear, M.: Cross-cultural detection of depression from nonverbal behaviour. In: 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, vol. 1, pp. 1–8 (2015)

    Google Scholar 

  28. Shi, J., Liu, C., Toshinori Ishi, C., Ishiguro, H.: Skeleton-based emotion recognition based on two-stream self-attention enhanced spatial-temporal graph convolutional network. Sensors 21, 205 (2021)

    Article  Google Scholar 

  29. Arunnehru, J., Geetha, K.: Automatic human emotion recognition in surveillance video. In: Intelligent Techniques in Signal Processing for Multimedia Security, pp. 321–342 (2017)

    Google Scholar 

  30. Sebastian-Kaltwang, M., Romera-Paredes, B., Martinez, B., Singh, A., Cella, M., Valstar, M.: The automatic detection of chronic pain-related expression: requirements, challenges and the multimodal emopain dataset. IEEE Trans. Affect. Comput. 7, 435–451 (2015)

    Google Scholar 

  31. Wang, W., Enescu, V., Sahli, H.: Adaptive real-time emotion recognition from body movements. ACM Trans. Interact. Intell. Syst. 5, 1–21 (2015)

    Google Scholar 

  32. Ahmed, F., Sieu, B., Gavrilova, M.: Score and rank-level fusion for emotion recognition using genetic algorithm. In: IEEE 17th International Conference on Cognitive Informatics & Cognitive Computing, pp. 46–53 (2018)

    Google Scholar 

  33. Wei, G., Jian, L., Mo, S.: Multimodal (audio, facial and gesture) based emotion recognition challenge. In: 15th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 908–911 (2020)

    Google Scholar 

  34. Yuan, X., Mahmoud, M.: Alanet: autoencoder-lstm for pain and protective behaviour detection. In: 15th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 824–828 (2020)

    Google Scholar 

  35. Egede, J., et al.: Emopain challenge 2020: multimodal pain evaluation from facial and bodily expressions. In: 15th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 849–856 (2020)

    Google Scholar 

  36. Gunes, H., Piccard, M.: A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In: 18th IEEE International Conference on Pattern Recognition, vol. 1, pp. 1148–115 (2006)

    Google Scholar 

  37. Douglas-Cowie, E., et al.:‘The humaine database. Emotion-Oriented Systems, pp. 243–284 (2011)

    Google Scholar 

  38. Baveye, Y., Dellandrea, E., Chamaret, C., Chen, L.: Liris-accede: a video database for affective content analysis. IEEE Trans. Affect. Comput. 6, 43–55 (2015)

    Article  Google Scholar 

  39. Bänziger, T., Mortillaro, M., Scherer, K.: Introducing the Geneva multimodal expression corpus for experimental research on emotion perception. Emotion 12, 1161 (2012)

    Article  Google Scholar 

  40. Fourati, N., Pelachaud, C.: Emilya: emotional body expression in daily actions database. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation, pp. 3486–3493 (2014)

    Google Scholar 

  41. Ma, Y., Paterson, H., Pollick, F.: A motion capture library for the study of identity, gender, and emotion perception from biological motion. Behav. Res. Methods 38, 134–141 (2006)

    Article  Google Scholar 

  42. Kleinsmith, A., De Silva, R., Bianchi-Berthouze, N.: Cross-cultural differences in recognizing affect from body posture. Interact. Comput. 18, 1371–1389 (2006)

    Article  Google Scholar 

  43. Emotional Body Motion Database. http://ebmdb.tuebingen.mpg.de/

  44. Barros, P., Churamani, N., Lakomkin, E., Siqueira, H., Sutherland, A., Wermter, S.: The omg-emotion behavior dataset. IEEE International Joint Conference on Neural Networks, pp. 1–7 (2018)

    Google Scholar 

  45. Keefe, B., Villing, M., Racey, C., Strong, S., Wincenciak, J., Barraclough, N.: A database of whole-body action videos for the study of action, emotion, and untrustworthiness. Behav. Res. Methods 46, 1042–1051 (2014)

    Article  Google Scholar 

  46. De Gelder, B., Van den Stock, J.: The bodily expressive action stimulus test (beast). Construction and validation of a stimulus basis for measuring perception of whole body expression of emotions. Front. Psychol. 2, 181 (2011)

    Article  Google Scholar 

  47. Sapiński, T., Kamińska, D., Pelikant, A., Ozcinar, C., Avots, E., Anbarjafari, G.: Multimodal database of emotional speech, video and gestures. In: International Conference on Pattern Recognition, pp. 153–163 (2018)

    Google Scholar 

  48. Busso, C., Bulut, M., Lee, C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J., Lee, S., Narayanan, S.: Iemocap: interactive emotional dyadic motion capture database. Lang. Resour. Eval. 42, 335–359 (2008)

    Article  Google Scholar 

  49. Volkova, E., De La Rosa, S., Bülthoff, H., Mohler, B.: The MPI emotional body expressions database for narrative scenarios. PLoS ONE 9, e113647 (2014)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Victoria, 3800 Finnerty Road, Victoria, BC, Canada

    Laleh Ebdali Takalloo & Kin Fun Li

  2. Kanagawa Institute of Technology, Atsugi, Kanagawa, Japan

    Kosuke Takano

Authors
  1. Laleh Ebdali Takalloo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kin Fun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kosuke Takano
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kin Fun Li .

Editor information

Editors and Affiliations

  1. Department of Information and Communication Engineering, Fukuoka Institute of Technology, Fukuoka, Japan

    Leonard Barolli

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ebdali Takalloo, L., Li, K.F., Takano, K. (2022). An Overview of Emotion Recognition from Body Movement. In: Barolli, L. (eds) Complex, Intelligent and Software Intensive Systems. CISIS 2022. Lecture Notes in Networks and Systems, vol 497. Springer, Cham. https://doi.org/10.1007/978-3-031-08812-4_11

Download citation

Publish with us

Policies and ethics