Skip to main content
SpringerLink
Log in

Cloud-oriented emotion feedback-based Exergames framework

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Exergames platform can be more appealing to the users if they can interact with emotion. Therefore, in this paper, we propose an automatic emotion recognition system from speech to be embedded in the Exergames platform. While playing and doing exercise, the user expresses his or her feeling by uttering some phrases. The speech is recorded by an omnidirectional mic and transmitted to an emotion recognition server in a cloud environment, where the emotion (e.g. happy, sad or neutral) is recognized. For the recognition, we use MPEG-7 low-level audio features and a Gaussian mixture model based classifier. A tactile vibration is generated based on the emotion and feedback to the user for a real feeling. The user can thus have an instantaneous vibrational feeling based on his or her satisfaction. The recognized emotion can also be used as the user’s satisfaction of the framework on the fly without the need of a survey session. The experimental study and performance comparison show that the proposed framework has positive effects on the perception of physical activities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Alamri, A Hassan, MM Hossain, MA Al-Qurishi, M Aldukhayyil, Y Hossain, M.S. (2014) “Evaluating the impact of a cloud-based serious game on obese people,” Computers in Human Behavior, Vol. 30, pp. 468–475

  2. Basori, AH Daman, D Bade, A Sunar, MS and Saari N (2008) “The feasibility of human haptic emotion as a feature to enhance interactivity and immersiveness on virtual reality game,” In Proc. VRCAI '08, Singapore

  3. Burkhardt, F Paeschke, A Rolfes, M Sendlmeier W and Weiss B (2005) “A database of German emotional speech”, Proc. Interspeech, 1517-1520, Lisbon, Portugal

  4. Chen M, Zhang Y, Li Y, Mao S, Leung V (2015) EMC: emotion-aware mobile cloud computing in 5G. IEEE Netw 29(2):32–38

    Article  Google Scholar 

  5. Chen M, Ma Y, Hao Y, Li Y, Wu D, Zhang Y, Song E (2016) “CP-robot: cloud-assisted pillow robot for emotion sensing and interaction”, Industrialiot 2016. Guangzhou, China

    Google Scholar 

  6. Ciota Z (2007), “Audio-haptic feedback in speech processing”, IEEE Intl. Workshop on Haptic Audio Visual Environments and their Applications (HAVE), pp. 67–70.

  7. Ginsburg KR (2007) The importance of play in promoting healthy child development and maintaining strong parent-child bonds. Pediatrics 119(1):182–191

    Article  Google Scholar 

  8. Hassan MM, Hossain MA, Alamri A, Hossain MS, Yaldukhayyil Y, Al-Qurishi M (2013) Health Monitoring of Obese People Through a Cloud-based Serious Game Framework. in Proc. IEEE ICME, San Jose

    Book  Google Scholar 

  9. Hossain MS, Muhammad G (2016) Audio-visual emotion recognition using multi-directional regression and Ridgelet transform. Springer J Multimodal User Interfaces 10(4):325–333

    Article  Google Scholar 

  10. Hossain; M. S. Muhammad, G. "An emotion recognition system for mobile applications," IEEE Access, vol.PP, no.99, pp.1–1, doi:10.1109/ACCESS.2017.2672829

  11. Hossain MS, Hassan MM, Alamri A (2013) An Exergames Framework for Obesity Monitoring and Management, in Proc. IEEE HAVE’2013, Istanbul

    Google Scholar 

  12. Hossain MS, Muhammad G, Song B, Hassan MM, Alelaiwi A, Alamri A (2015) Audio–visual emotion-aware cloud gaming framework. IEEE Trans Circuits Syst Video Technol 25(12):2105–2118

    Article  Google Scholar 

  13. Hossain MS, Hardy S, Alamri A, Alelawi A, Hardy V, Wilhelm C (2016) AR-based serious games framework for post-stroke rehabilitation. ACM/Springer Multimed Syst 22(6):659–674

    Article  Google Scholar 

  14. Hossain MS, Muhammad G, Al Hamid MF, Song B (2016) Audio-visual emotion-aware big data recognition towards 5G. Mob Networks Appl 21(5):753–763

    Article  Google Scholar 

  15. Information Technology-Multimedia Content Description Interface-Part 4: Audio, ISO/IEC CD 15938–4, 2001

  16. Karime A et al (2012) Tele-Wobble: A Telerehabilitation Wobble Board for Lower Extremity Therapy. IEEE Trans Instrum Meas 61(7):1816–1824

    Article  Google Scholar 

  17. Koolagudi SG, Rao KS (June 2012) Emotion recognition from speech: a review. Intern J Speech Technol 15(2):99–117

    Article  Google Scholar 

  18. Mateas, M et al. (2007) “presence and engagement in an interactive Drama”, in Proc. CHI’07 conference on computer human interaction (CHI), San Jose, CA, USA.

  19. Muhammad G and Alghathbar K (2009), “Environment recognition from audio using MPEG-7 features,” Proc. The 4th International Conference on Embedded and Multimedia Computing (EM-Com09), S. Korea.

  20. Muhammad G, Alghathbar K (2011) Environment recognition for digital audio forensics using MPEG–7 and Mel cepstral features. J Electr Eng 62(4):199–205

    Google Scholar 

  21. Parker JR, Baradoy G, Katz L Using virtual reality technology and biometric interfaces in obesity reduction. Can J Diabetes 35(2):187

  22. Patsi C. et al. (2012) “Exergamess and their Effect on Emotional State in People with Schizophrenia “ Balkan Military Medical Review 15(4):275–281

  23. Schuller, B Rigoll, G and Lang M (2004) “Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine - belief network architecture”, International Conf. Acoustics, Speech, and Signal Processing (ICASSP), pp. I-577-580.

  24. Sinclair, J Hingston, P and Masek M (2007) “Considerations for the design of Exergamess,” In Proc. ACM GRAPHITE’07, Perth, Australia

  25. Tadeusz Stach and T.C. Nicholas Graham (2011) “Exploring Haptic Feedback in Exergamess,” P. Campos et al. (Eds.): INTERACT 2011, Part II, LNCS 6947, pp. 18–35

  26. Szczuko, P Dalka, P Dabrowski, M and Kostek B (2004) “MPEG-7-based Low-Level Descriptor Effectiveness in the Automatic Musical Sound Classification”, Proc. AES 116th Convention.

  27. TU-Berlin MPEG-7 Audio Analyzer. http://mpeg7lld.nue.tu-berlin.de/.

  28. Ververidis D, Kotropoulos C (2006) Emotional speech recognition: resources, features, and methods. Speech Comm 48(9):1162–1181

    Article  Google Scholar 

  29. Warburton D, Bredin S, Horita L, Zbogar D, Scott J, Esch B et al (2007) The health benefits of interactive video game exercise. Appl Physiol Nutr Metab 32(4):655–663

    Article  Google Scholar 

  30. WHO Obesity and overweight fact sheet. http://www.who.int/mediacentre/factsheets/fs311/en/index.html Retrieved Oct 2016

  31. Yim J, Graham TCN (2007) Using games to increase exercise motivation. Proc. Future Play, In, pp 166–173

    Google Scholar 

  32. Zacharatos H (2013) Emotion recognition for Exergamess using Laban movement analysis. In Proc. ACM SIGGRAPH conference on Motion in Games, Dublin, UK

    Google Scholar 

  33. Zhang Y (2016) Grorec: a group-centric intelligent recommender system integrating social, mobile and big data technologies. IEEE Trans Serv Comput 9(5):786–795

    Article  Google Scholar 

  34. Zhang Y et al (2017) iDoctor: personalized and professionalized medical recommendations based on hybrid matrix factorization. Futur Gener Comput Syst 66:30–35

    Article  Google Scholar 

  35. Y Zhou, Y Sun, J Zhang, and Y Yan (2009) "Speech Emotion Recognition Using Both Spectral and Prosodic Features", International Conf. Information Engineering and Computer Science (ICIECS), 1–4.

Download references

Acknowledgments

This work is financially supported by the King Saud University, Deanship of Scientific Research, Research Chair of Pervasive and Mobile Computing.

Author information

Authors and Affiliations

  1. Department of Software Engineering, College of Computer and Information Sciences (CCIS), King Saud University, P.O.Box 51178, Riyadh, 11543, Saudi Arabia

    M. Shamim Hossain & Atif Alamri

  2. Department of Computer Engineering, College of Computer and Information Sciences (CCIS), King Saud University, P.O.Box 51178, Riyadh, 11543, Saudi Arabia

    Ghulam Muhammad & Wadood Abdul

  3. Research Chair of Pervasive and Mobile Computing, King Saud University, Riyadh, 11543, Saudi Arabia

    Muhammad Al-Qurishi, Ahmad Almogren, Wadood Abdul & Atif Alamri

  4. Department of Computer Science, Taif University, Taif, Saudi Arabia

    Mehedi Masud

Authors
  1. M. Shamim Hossain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ghulam Muhammad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muhammad Al-Qurishi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mehedi Masud
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ahmad Almogren
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wadood Abdul
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Atif Alamri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. Shamim Hossain.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shamim Hossain, M., Muhammad, G., Al-Qurishi, M. et al. Cloud-oriented emotion feedback-based Exergames framework. Multimed Tools Appl 77, 21861–21877 (2018). https://doi.org/10.1007/s11042-017-4621-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4621-1

Keywords

Search

Navigation