Skip to main content
SpringerLink
Log in

Progress in Multimodal Affective Computing: From Machine Learning to Deep Learning

  • Chapter
  • First Online:
Smart Computer Vision

Part of the book series: EAI/Springer Innovations in Communication and Computing ((EAISICC))

  • 506 Accesses

Abstract

Introduction: Human reaction varies from person to person, so developing a generic model to find the emotion has its own requirement. Due to the increase in the necessity to interpret the emotions in various sectors like medical, educational, emotional interference, etc., multimodal affective computing plays a vital role. Enhancements in the domain of human-computer interaction have led to the progression from unimodal to multimodal, which have gained the interest of research society around the globe. One of the major reasons for migrating from unimodal to multimodal is the improvement in the performance of affect recognition.

Objectives: The main objective of this chapter is to perform a comprehensive survey on most active research done in the field of machine learning and deep learning for multimodal affective computing.

Contribution: This chapter will cover various information of publicly available dataset with the modalities and the emotions in each dataset. This study will provide details of commonly used features and fusion techniques. Various machine learning and deep learning techniques for affect recognition were explained. Further, we identified and discussed a set of real-word applications where affect recognition using machine learning and deep learning techniques is required.

Methods, Results, and Conclusion: In this chapter, we have discussed a general overview of affective computing and multimodal affective computing, types of features, fusion techniques, information about dataset like modality, type of data and emotions, and models used for machine learning- and deep learning-based techniques and also focus on various research areas of machine learning and deep learning for multimodal affect recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ayata, D., Yaslan, Y., & Kamasak, M. E. (2018). Emotion based music recommendation system using wearable physiological sensors. IEEE Transactions on Consumer Electronics, 64(2), 196–203.

    Article  Google Scholar 

  2. Bălan, O., Moise, G., Moldoveanu, A., Leordeanu, M., & Moldoveanu, F. (2020). An investigation of various machine and deep learning techniques applied in automatic fear level detection and acrophobia virtual therapy. Sensors, 20(2), 496.

    Article  Google Scholar 

  3. Chakravarthula, S. N., Nasir, M., Tseng, S. Y., Li, H., Park, T. J., Baucom, B., et al. (2020, May). Automatic prediction of suicidal risk in military couples using multimodal interaction cues from couples conversations. In ICASSP 2020–2020 IEEE international conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6539–6543). IEEE.

    Google Scholar 

  4. Comas, J., Aspandi, D., & Binefa, X. (2020, November). End-to-end facial and physiological model for affective computing and applications. In 2020 15th IEEE international conference on Automatic Face and Gesture Recognition (FG 2020) (pp. 93–100). IEEE.

    Google Scholar 

  5. Garcia-Garcia, J. M., Penichet, V. M., Lozano, M. D., Garrido, J. E., & Law, E. L. C. (2018). Multimodal affective computing to enhance the user experience of educational software applications. Mobile Inf Syst, 2018.

    Google Scholar 

  6. Henderson, N. L., Rowe, J. P., Mott, B. W., & Lester, J. C. (2019). Sensor-based data fusion for multimodal affect detection in game-based learning environments. In EDM (workshops) (pp. 44–50).

    Google Scholar 

  7. Huang, J., Li, Y., Tao, J., Lian, Z., Niu, M., & Yang, M. (2018, October). Multimodal continuous emotion recognition with data augmentation using recurrent neural networks. In Proceedings of the 2018 on audio/visual emotion challenge and workshop (pp. 57–64).

    Google Scholar 

  8. Huang, Y., Yang, J., Liao, P., & Pan, J. (2017). Fusion of facial expressions and EEG for multimodal emotion recognition. Computational Intelligence and Neuroscience, 2017, 1.

    Google Scholar 

  9. Jan, A., Meng, H., Gaus, Y. F. B. A., & Zhang, F. (2017). Artificial intelligent system for automatic depression level analysis through visual and vocal expressions. IEEE Transactions on Cognitive and Developmental Systems, 10(3), 668–680.

    Article  Google Scholar 

  10. Jang, E. H., Byun, S., Park, M. S., & Sohn, J. H. (2020). Predicting individuals’ experienced fear from multimodal physiological responses to a fear-inducing stimulus. Advances in Cognitive Psychology, 16(4), 291.

    Article  Google Scholar 

  11. Jung, T. P., & Sejnowski, T. J. (2018, July). Multi-modal approach for affective computing. In 2018 40th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (pp. 291–294). IEEE.

    Google Scholar 

  12. Kim, E., & Shin, J. W. (2019, May). Dnn-based emotion recognition based on bottleneck acoustic features and lexical features. In ICASSP 2019-2019 IEEE international conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6720–6724). IEEE.

    Google Scholar 

  13. Kim, S. K., & Kang, H. B. (2018). An analysis of fear of crime using multimodal measurement. Biomedical Signal Processing and Control, 41, 186–197.

    Article  Google Scholar 

  14. Li, Y., Tao, J., Schuller, B., Shan, S., Jiang, D., & Jia, J. (2018, May). Mec 2017: Multimodal emotion recognition challenge. In 2018 first Asian conference on Affective Computing and Intelligent Interaction (ACII Asia) (pp. 1–5). IEEE.

    Google Scholar 

  15. Ma, J., Tang, H., Zheng, W. L., & Lu, B. L. (2019, October). Emotion recognition using multimodal residual LSTM network. In Proceedings of the 27th ACM International conference on multimedia (pp. 176–183).

    Google Scholar 

  16. Marín-Morales, J., Higuera-Trujillo, J. L., Greco, A., Guixeres, J., Llinares, C., Scilingo, E. P., et al. (2018). Affective computing in virtual reality: Emotion recognition from brain and heartbeat dynamics using wearable sensors. Scientific Reports, 8(1), 1–15.

    Article  Google Scholar 

  17. Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020, April). M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 02, pp. 1359–1367).

    Google Scholar 

  18. Mou, L., Zhou, C., Zhao, P., Nakisa, B., Rastgoo, M. N., Jain, R., & Gao, W. (2021). Driver stress detection via multimodal fusion using attention-based CNN-LSTM. Expert Systems with Applications, 173, 114693.

    Article  Google Scholar 

  19. Muszynski, M., Tian, L., Lai, C., Moore, J., Kostoulas, T., Lombardo, P., et al. (2019). Recognizing induced emotions of movie audiences from multimodal information. IEEE Transactions on Affective Computing, 12, 36–52.

    Article  Google Scholar 

  20. Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V. (2019). A hybrid latent space data fusion method for multimodal emotion recognition. IEEE Access, 7, 172948–172964.

    Article  Google Scholar 

  21. Papakostas, M., Riani, K., Gasiorowski, A. B., Sun, Y., Abouelenien, M., Mihalcea, R., & Burzo, M. (2021, April). Understanding driving distractions: A multimodal analysis on distraction characterization. In 26th international conference on Intelligent User Interfaces (pp. 377–386).

    Google Scholar 

  22. Qureshi, S. A., Saha, S., Hasanuzzaman, M., & Dias, G. (2019). Multitask representation learning for multimodal estimation of depression level. IEEE Intelligent Systems, 34(5), 45–52.

    Article  Google Scholar 

  23. Ramakrishnan, A., Zylich, B., Ottmar, E., LoCasale-Crouch, J., & Whitehill, J. (2021). Toward automated classroom observation: Multimodal machine learning to estimate class positive climate and negative climate. IEEE Transactions on Affective Computing.

    Google Scholar 

  24. Shin, D., Shin, D., & Shin, D. (2017). Development of emotion recognition interface using complex EEG/ECG bio-signal for interactive contents. Multimedia Tools and Applications, 76(9), 11449–11470.

    Article  Google Scholar 

  25. Tzirakis, P., Chen, J., Zafeiriou, S., & Schuller, B. (2021). End-to-end multimodal affect recognition in real-world environments. Information Fusion, 68, 46–53.

    Article  Google Scholar 

  26. Tzirakis, P., Trigeorgis, G., Nicolaou, M. A., Schuller, B. W., & Zafeiriou, S. (2017). End-to-end multimodal emotion recognition using deep neural networks. IEEE Journal of Selected Topics in Signal Processing, 11(8), 1301–1309.

    Article  Google Scholar 

  27. Wang, C. H., & Lin, H. C. K. (2018). Emotional design tutoring system based on multimodal affective computing techniques. International Journal of Distance Education Technologies (IJDET), 16(1), 103–117.

    Article  Google Scholar 

  28. Yoon, S., Byun, S., & Jung, K. (2018, December). Multimodal speech emotion recognition using audio and text. In 2018 IEEE Spoken Language Technology Workshop (SLT) (pp. 112–118). IEEE.

    Google Scholar 

  29. Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2017). Learning affective features with a hybrid deep model for audio–visual emotion recognition. IEEE Transactions on Circuits and Systems for Video Technology, 28(10), 3030–3043.

    Article  Google Scholar 

  30. Zheng, W. L., Liu, W., Lu, Y., Lu, B. L., & Cichocki, A. (2018). Emotionmeter: A multimodal framework for recognizing human emotions. IEEE transactions on cybernetics, 49(3), 1110–1122.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Amrita School of Engineering, Coimbatore, Amrita Vishwa Vidyapeetham, Coimbatore, India

    M. Chanchal

  2. Department of Information Technology, PSG College of Technology, Coimbatore, India

    B. Vinoth Kumar

Authors
  1. M. Chanchal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. B. Vinoth Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. PSG College of Technology, Coimbatore, Tamil Nadu, India

    B. Vinoth Kumar

  2. Thiruvettakudy, National Institute of Technology, Puducherry, Karaikal, India

    P. Sivakumar

  3. Thiruvettakudy, National Institute of Technology, Puducherry, Karaikal, India

    B. Surendiran

  4. University of North Texas, Denton, TX, USA

    Junhua Ding

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Chanchal, M., Vinoth Kumar, B. (2023). Progress in Multimodal Affective Computing: From Machine Learning to Deep Learning. In: Kumar, B.V., Sivakumar, P., Surendiran, B., Ding, J. (eds) Smart Computer Vision. EAI/Springer Innovations in Communication and Computing. Springer, Cham. https://doi.org/10.1007/978-3-031-20541-5_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20541-5_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20540-8

  • Online ISBN: 978-3-031-20541-5

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation