Skip to main content
SpringerLink
Log in

Audio-Visual Emotion Recognition Using Big Data Towards 5G

  • Published:
Mobile Networks and Applications Aims and scope Submit manuscript

Abstract

With the advent of future generation mobile communication technologies (5G), there is the potential to allow mobile users to have access to big data processing over different clouds and networks. The increasing numbers of mobile users come with additional expectations for personalized services (e.g., social networking, smart home, health monitoring) at any time, from anywhere, and through any means of connectivity. Because of the expected massive amount of complex data generated by such services and networks from heterogeneous multiple sources, an infrastructure is required to recognize a user’s sentiments (e.g., emotion) and behavioral patterns to provide a high quality mobile user experience. To this end, this paper proposes an infrastructure that combines the potential of emotion-aware big data and cloud technology towards 5G. With this proposed infrastructure, a bimodal system of big data emotion recognition is proposed, where the modalities consist of speech and face video. Experimental results show that the proposed approach achieves 83.10 % emotion recognition accuracy using bimodal inputs. To show the suitability and validity of the proposed approach, Hadoop-based distributed processing is used to speed up the processing for heterogeneous mobile clients.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Hossain E, Hasan M (2015) 5G cellular: key enabling technologies and research challenges. IEEE Instrum Meas Mag 18(3):11–21

    Article  Google Scholar 

  2. Han Q, Liang S, Zhang H (2015) Mobile cloud sensing, big data, and 5G networks make an intelligent and smart world. IEEE Netw 29(2):40–45

    Article  Google Scholar 

  3. Chen M, Mao S, Li Y, Mao S (2014) Big data: a survey. ACM/Springer Mobile Networks and Applications 19(2):171–209

    Article  Google Scholar 

  4. Baimbetov Y, Khalil I, Steinbauer M, Anderst-Kotsis G (2015) Using big data for emotionally intelligent mobile services through multi-modal emotion recognition. In: Inclusive smart cities and e-health:lecture notes in computer science, vol 9102. Springer, pp 127–138

  5. Hossain MS, Muhammad G, Song B, Hassan M, Alelaiwi A, Alamri A (2015) Audio-visual emotion-aware cloud gaming framework. IEEE Trans Circuits Syst Video Technol 25(12):2105–2118

    Article  Google Scholar 

  6. Chen M, Hao Y, Li Y, Wu D, Huang D (2015) Demo: LIVES: Learning through interactive video and emotion aware system. In: ACM Mobihoc 2015. Hangzhou, pp 22–25

  7. Hossain MS, Muhammad G (2015) Cloud-assisted speech and face recognition framework for health monitoring. ACM/Springer Mobile Networks and Applications 20(3):391–399

    Article  Google Scholar 

  8. Chen M, Zhang Y, Li Y, Mao S, Leung V (2015) EMC: Emotion-aware mobile cloud computing in 5G. IEEE Netw 29(2):32–38

    Article  Google Scholar 

  9. Chen M, Zhang Y, Li Y, Hassan M, Alamri A (2015) AIWAC: Affective interaction through wearable computing and cloud technology. IEEE Wirel Commun Mag 22(1):20–27

    Article  Google Scholar 

  10. Hossain MS, Muhammad G (2015) Audio-visual emotion recognition using multi-directional regression and Ridgelet transform. Springer J. Multimodal User Interfaces

  11. Chen M (2014) NDNC-BAN: supporting rich media healthcare services via named data networking in cloud-assisted wireless body area networks. Inf Sci 284(10):142–156

    Article  Google Scholar 

  12. Schuller B, Rigoll G, Lang M (2004) Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine belief network architecture. In: IEEE ICASSP’04

  13. Zhou Y, Sun Y, Zhang J, Yan Y (2009) Speech emotion recognition using both spectral and prosodic features. In: ICIECS’09

  14. Albornoz EM, Milone DH, Rufiner HL (2011) Spoken emotion recognition using hierarchical classifiers. Comput Speech Lang 25:556–570

    Article  Google Scholar 

  15. Burkhardt F, Paeschke A, Rolfes M, Sendlmeier W, Weiss B (2005) A database of German emotional speech. In: Interspeech’2005, Lisbon, Portugal

  16. Bettadapura V (2012) Face expression recognition and analysis: the state of the art. College of Computing, Georgia Institute of Technology

  17. Senechal T, Rapp V, Salam H, Seguier R, Bailly K, Prevost L (2012) Facial action recognition combining heterogeneous features via multikernel learning. IEEE Trans Syst Man Cybern B Cybern 42(4):993–1005

    Article  Google Scholar 

  18. Majumder A, Behera L, Subramanian VK (2014) Emotion recognition from geometric facial features using self organizing map. Pattern Recogn 47(3):1282–1293

    Article  Google Scholar 

  19. Bejani M, Gharavian D, Charkari NM (2014) Audiovisual emotion recognition using ANOVA feature selection method and multi classifier neural networks. Neural Comput & Applic 24(2):399–412

    Article  Google Scholar 

  20. Martin O, Kotsia I, Macq B, Pitas I (2006) The eNTERFACE’05 audiovisual emotion database. In: ICDEW’2006, Atlanta, GA

  21. Kachele M, Glodek M, Zharkov D, Meudt S, Schwenker F (2014) Fusion of audio-visual features using hierarchical classifier systems for the recognition of affective states and the state of depression. In: ICPRAM’14

  22. Jeremie N, Vincent R, Kevin B, Lionel P, Mohamed C (2014) Audio-visual emotion recognition: A dynamic, multimodal approach. In: IHM’14, Lille, France

  23. Ryu C, Lee D, Jang M, Kim C, Seo E (2013) Extensible video processing framework in apache Hadoop. In: IEEE International conference on cloud computing technology and science, vol 2, pp 305–310

  24. Wang H, et al. (2012) Large-scale multimedia data mining using MapReduce framework. In: IEEE CloudCom, pp 287–292

  25. Apache Hadoop. http://hadoop.apache.org/

  26. Tan H, Chen L (2014) An approach for fast and parallel video processing on Apache Hadoop clusters. In: IEEE ICME

  27. Kim M, Han S, Cui Y, Lee H, Cho H, Hwang S (2014) CloudDMSS: robust Hadoop-based multimedia streaming service architecture for a cloud computing environment. Clust Comput 17(3):1386–7857

    Google Scholar 

  28. Chiu O Microsoft delivers interactive analytics on Big Data with the release of Spark for Azure HDInsight. https://azure.microsoft.com/en-us/blog/interactive-analytics-on-big-data-with-the-release-of-spark-for-azure-hdinsight/

  29. FFMPEG. https://www.ffmpeg.org/

  30. Open Source Computer Visiopn (OpenCV). http://opencv.org/

  31. Muhammad G, Mesallam T, Almalki K, Farahat M, Mahmood A, Alsulaiman M (2012) Multi Directional Regression (MDR) Based Features for Automatic Voice Disorder Detection. J Voice 26(6):817.e19–817.e27

    Article  Google Scholar 

  32. Dollar P, Rabaud V, Cottrell G, Belongie S (2005) Behavior recognition via sparse spatio-temporal features. In: IEEE VS-PETS’05, Beijing, China

  33. Kanade T, Cohn J, Tian Y (2000) Comprehensive database for facial expression analysis. In: IEEE AFGR’00

  34. Muhammad G, Masud M, Alelaiwi A, Rahman MA, Karime A, Alamri A, Hossain MS (2015) Spectro-temporal directional derivative based automatic speech recognition for a serious game scenario. Multimedia Tools and Applications 74(14):5313–5327

    Article  Google Scholar 

  35. Chen J, Shan S, He C, Zhao G, Pietikainen M, Chen X, Gao W (2010) WLD: a robust local image descriptor. IEEE Trans Pattern Anal Mach Intell 32(9):1705–1720

    Article  Google Scholar 

  36. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20:273–297

    MATH  Google Scholar 

  37. Kim M, Cui Y, Han S, Lee HP (2013) Towards efficient design and implementation of a Hadoop-based distributed video transcoding system in cloud computing environment. J Multimed Ubiquitous Eng 8(2):213–224

    Google Scholar 

Download references

Acknowledgments

The authors extend their appreciation to the Deanship of Scientific Research at King Saud University, Riyadh, Saudi Arabia for funding this work through the research group project no. RGP-1436-023.

Author information

Authors and Affiliations

  1. Software Engineering Department, College of Computer and Information Sciences (CCIS), King Saud University, Riyadh, 11543, Kingdom of Saudi Arabia

    M. Shamim Hossain, Mohammed F. Alhamid & Khaled Al-Mutib

  2. Computer Engineering Department, College of Computer and Information Sciences (CCIS), King Saud University, Riyadh, 11543, Kingdom of Saudi Arabia

    Ghulam Muhammad

  3. Information System Department, College of Computer and Information Sciences (CCIS), King Saud University, Riyadh, 11543, Kingdom of Saudi Arabia

    Biao Song

Authors
  1. M. Shamim Hossain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ghulam Muhammad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammed F. Alhamid
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Biao Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Khaled Al-Mutib
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. Shamim Hossain.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hossain, M.S., Muhammad, G., Alhamid, M.F. et al. Audio-Visual Emotion Recognition Using Big Data Towards 5G. Mobile Netw Appl 21, 753–763 (2016). https://doi.org/10.1007/s11036-016-0685-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11036-016-0685-9

Keywords

Search

Navigation