Abstract
Exergames platform can be more appealing to the users if they can interact with emotion. Therefore, in this paper, we propose an automatic emotion recognition system from speech to be embedded in the Exergames platform. While playing and doing exercise, the user expresses his or her feeling by uttering some phrases. The speech is recorded by an omnidirectional mic and transmitted to an emotion recognition server in a cloud environment, where the emotion (e.g. happy, sad or neutral) is recognized. For the recognition, we use MPEG-7 low-level audio features and a Gaussian mixture model based classifier. A tactile vibration is generated based on the emotion and feedback to the user for a real feeling. The user can thus have an instantaneous vibrational feeling based on his or her satisfaction. The recognized emotion can also be used as the user’s satisfaction of the framework on the fly without the need of a survey session. The experimental study and performance comparison show that the proposed framework has positive effects on the perception of physical activities.
Similar content being viewed by others
References
Alamri, A Hassan, MM Hossain, MA Al-Qurishi, M Aldukhayyil, Y Hossain, M.S. (2014) “Evaluating the impact of a cloud-based serious game on obese people,” Computers in Human Behavior, Vol. 30, pp. 468–475
Basori, AH Daman, D Bade, A Sunar, MS and Saari N (2008) “The feasibility of human haptic emotion as a feature to enhance interactivity and immersiveness on virtual reality game,” In Proc. VRCAI '08, Singapore
Burkhardt, F Paeschke, A Rolfes, M Sendlmeier W and Weiss B (2005) “A database of German emotional speech”, Proc. Interspeech, 1517-1520, Lisbon, Portugal
Chen M, Zhang Y, Li Y, Mao S, Leung V (2015) EMC: emotion-aware mobile cloud computing in 5G. IEEE Netw 29(2):32–38
Chen M, Ma Y, Hao Y, Li Y, Wu D, Zhang Y, Song E (2016) “CP-robot: cloud-assisted pillow robot for emotion sensing and interaction”, Industrialiot 2016. Guangzhou, China
Ciota Z (2007), “Audio-haptic feedback in speech processing”, IEEE Intl. Workshop on Haptic Audio Visual Environments and their Applications (HAVE), pp. 67–70.
Ginsburg KR (2007) The importance of play in promoting healthy child development and maintaining strong parent-child bonds. Pediatrics 119(1):182–191
Hassan MM, Hossain MA, Alamri A, Hossain MS, Yaldukhayyil Y, Al-Qurishi M (2013) Health Monitoring of Obese People Through a Cloud-based Serious Game Framework. in Proc. IEEE ICME, San Jose
Hossain MS, Muhammad G (2016) Audio-visual emotion recognition using multi-directional regression and Ridgelet transform. Springer J Multimodal User Interfaces 10(4):325–333
Hossain; M. S. Muhammad, G. "An emotion recognition system for mobile applications," IEEE Access, vol.PP, no.99, pp.1–1, doi:10.1109/ACCESS.2017.2672829
Hossain MS, Hassan MM, Alamri A (2013) An Exergames Framework for Obesity Monitoring and Management, in Proc. IEEE HAVE’2013, Istanbul
Hossain MS, Muhammad G, Song B, Hassan MM, Alelaiwi A, Alamri A (2015) Audio–visual emotion-aware cloud gaming framework. IEEE Trans Circuits Syst Video Technol 25(12):2105–2118
Hossain MS, Hardy S, Alamri A, Alelawi A, Hardy V, Wilhelm C (2016) AR-based serious games framework for post-stroke rehabilitation. ACM/Springer Multimed Syst 22(6):659–674
Hossain MS, Muhammad G, Al Hamid MF, Song B (2016) Audio-visual emotion-aware big data recognition towards 5G. Mob Networks Appl 21(5):753–763
Information Technology-Multimedia Content Description Interface-Part 4: Audio, ISO/IEC CD 15938–4, 2001
Karime A et al (2012) Tele-Wobble: A Telerehabilitation Wobble Board for Lower Extremity Therapy. IEEE Trans Instrum Meas 61(7):1816–1824
Koolagudi SG, Rao KS (June 2012) Emotion recognition from speech: a review. Intern J Speech Technol 15(2):99–117
Mateas, M et al. (2007) “presence and engagement in an interactive Drama”, in Proc. CHI’07 conference on computer human interaction (CHI), San Jose, CA, USA.
Muhammad G and Alghathbar K (2009), “Environment recognition from audio using MPEG-7 features,” Proc. The 4th International Conference on Embedded and Multimedia Computing (EM-Com09), S. Korea.
Muhammad G, Alghathbar K (2011) Environment recognition for digital audio forensics using MPEG–7 and Mel cepstral features. J Electr Eng 62(4):199–205
Parker JR, Baradoy G, Katz L Using virtual reality technology and biometric interfaces in obesity reduction. Can J Diabetes 35(2):187
Patsi C. et al. (2012) “Exergamess and their Effect on Emotional State in People with Schizophrenia “ Balkan Military Medical Review 15(4):275–281
Schuller, B Rigoll, G and Lang M (2004) “Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine - belief network architecture”, International Conf. Acoustics, Speech, and Signal Processing (ICASSP), pp. I-577-580.
Sinclair, J Hingston, P and Masek M (2007) “Considerations for the design of Exergamess,” In Proc. ACM GRAPHITE’07, Perth, Australia
Tadeusz Stach and T.C. Nicholas Graham (2011) “Exploring Haptic Feedback in Exergamess,” P. Campos et al. (Eds.): INTERACT 2011, Part II, LNCS 6947, pp. 18–35
Szczuko, P Dalka, P Dabrowski, M and Kostek B (2004) “MPEG-7-based Low-Level Descriptor Effectiveness in the Automatic Musical Sound Classification”, Proc. AES 116th Convention.
TU-Berlin MPEG-7 Audio Analyzer. http://mpeg7lld.nue.tu-berlin.de/.
Ververidis D, Kotropoulos C (2006) Emotional speech recognition: resources, features, and methods. Speech Comm 48(9):1162–1181
Warburton D, Bredin S, Horita L, Zbogar D, Scott J, Esch B et al (2007) The health benefits of interactive video game exercise. Appl Physiol Nutr Metab 32(4):655–663
WHO Obesity and overweight fact sheet. http://www.who.int/mediacentre/factsheets/fs311/en/index.html Retrieved Oct 2016
Yim J, Graham TCN (2007) Using games to increase exercise motivation. Proc. Future Play, In, pp 166–173
Zacharatos H (2013) Emotion recognition for Exergamess using Laban movement analysis. In Proc. ACM SIGGRAPH conference on Motion in Games, Dublin, UK
Zhang Y (2016) Grorec: a group-centric intelligent recommender system integrating social, mobile and big data technologies. IEEE Trans Serv Comput 9(5):786–795
Zhang Y et al (2017) iDoctor: personalized and professionalized medical recommendations based on hybrid matrix factorization. Futur Gener Comput Syst 66:30–35
Y Zhou, Y Sun, J Zhang, and Y Yan (2009) "Speech Emotion Recognition Using Both Spectral and Prosodic Features", International Conf. Information Engineering and Computer Science (ICIECS), 1–4.
Acknowledgments
This work is financially supported by the King Saud University, Deanship of Scientific Research, Research Chair of Pervasive and Mobile Computing.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Shamim Hossain, M., Muhammad, G., Al-Qurishi, M. et al. Cloud-oriented emotion feedback-based Exergames framework. Multimed Tools Appl 77, 21861–21877 (2018). https://doi.org/10.1007/s11042-017-4621-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-017-4621-1