Skip to main content
Springer Nature Link
Log in

Emotional representation of music in multi-source data by the Internet of Things and deep learning

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

The increasing number of users on the internet and the massive digital music require efficient music retrieval means and a satisfactory retrieval experience for users. The objectives are to enable users to have a common emotional interaction with the emotional representation of the music itself in the process of experiencing music. Firstly, starting from the principle of multi-modal technology, Internet of Things sensors are used to recognize the emotional representation of music. Secondly, the Naive Bayes Classifier based on deep learning is used to classify the emotion of music works, as well as the emotion classification of users' feelings of music. Finally, the analysis of the emotional representation of music and emotional experience analysis of the user is carried out, and the accuracy of music emotion classification is also studied from different classification methods. The results show that under the Naive Bayes classification, the highest recognition rate of music in the emotionally exciting category is 0.53, the healing category is 0.32, the relaxation category is 0.16, the romance category is 0.30, the nostalgic category is 0.35, the loneliness category is 0.28, and the quiet category is 0.21. The highest recognition rate of loneliness in the user's emotions is 0.63, followed by nostalgia, excitement and romance, which are 0.55, 0.45 and 0.31, respectively. According to the analysis of user experience and the use of listening to songs, music representations such as loneliness are in line with user experience. Naive Bayes has the highest accuracy in music emotion classification in multi-source data, which is 86.64%. It has important reference significance for multimodal music emotional analysis and emotional resonance analysis between music and listeners.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

References

  1. Li C, Bao Z, Li L et al (2020) Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition. Inf Process Manag 57(3):102185. https://doi.org/10.1016/j.ipm.2019.102185

    Article  Google Scholar 

  2. Zhang X, Liu J, Shen J et al (2020) Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine. IEEE Trans Cybern 51(9):4386–4399. https://doi.org/10.1109/TCYB.2020.2987575

    Article  Google Scholar 

  3. Doma V, Pirouz M (2020) A comparative analysis of machine learning methods for emotion recognition using EEG and peripheral physiological signals. J Big Data 7(1):1–21. https://doi.org/10.1109/ISSC49989.2020.9180193

    Article  Google Scholar 

  4. Nogaj AA (2020) Emotional intelligence and strategies for coping with stress among music school students in the context of visual art and general education students. J Res Music Educ 68(1):78–96. https://doi.org/10.1177/0022429420901513

    Article  Google Scholar 

  5. Filippi P, Hoeschele M, Spierings M et al (2019) Temporal modulation in speech, music, and animal vocal communication: evidence of conserved function. Ann N Y Acad Sci 1453(1):99–113. https://doi.org/10.1111/nyas.14228

    Article  Google Scholar 

  6. Cohen S, Bodner E (2019) The relationship between flow and music performance anxiety amongst professional classical orchestral musicians. Psychol Music 47(3):420–435

    Article  Google Scholar 

  7. Küssner MB, Eerola T (2019) The content and functions of vivid and soothing visual imagery during music listening: findings from a survey study. Psychol Music Mind Brain 29(2–3):90. https://doi.org/10.1177/0305735618754689

    Article  Google Scholar 

  8. Schäfer K, Eerola T (2020) How listening to music and engagement with other media provide a sense of belonging: an exploratory study of social surrogacy. Psychol Music 48(2):232–251. https://doi.org/10.1177/0305735618795036

    Article  Google Scholar 

  9. Fancourt D, Garnett C, Spiro N et al (2019) How do artistic creative activities regulate our emotions? Validation of the Emotion Regulation Strategies for Artistic Creative Activities Scale (ERS-ACA). PLoS ONE 14(2):e0211362. https://doi.org/10.1371/journal.pone.0211362

    Article  Google Scholar 

  10. Dong Y, Yang X, Zhao X et al (2019) Bidirectional convolutional recurrent sparse network (BCRSN): an efficient model for music emotion recognition. IEEE Trans Multimed 21(12):3150–3163. https://doi.org/10.1109/TMM.2019.2918739

    Article  Google Scholar 

  11. Lin HCK, Ma YC, Lee M (2021) Constructing emotional machines: a case of a smartphone-based emotion system. Electronics 10(3):306. https://doi.org/10.3390/electronics10030306

    Article  Google Scholar 

  12. Darvariu VA, Convertino L, Mehrotra A et al (2020) Quantifying the relationships between everyday objects and emotional states through deep learning based image analysis using smartphones. Proc ACM Interact Mob Wearable Ubiquitous Technol 4(1):1–21. https://doi.org/10.1145/3380997

    Article  Google Scholar 

  13. Elhamdadi H, Canavan S, Rosen P (2021) AffectiveTDA: using topological data analysis to improve analysis and explainability in affective computing. IEEE Trans Vis Comput Graph 28(1):769–779. https://doi.org/10.1109/TVCG.2021.3114784

    Article  Google Scholar 

  14. Isohätälä J, Näykki P, Järvelä S (2020) Cognitive and socio-emotional interaction in collaborative learning: exploring fluctuations in students’ participation. Scand J Educ Res 64(6):831–851. https://doi.org/10.1080/00313831.2019.1623310

    Article  Google Scholar 

  15. Qiu J, Zhou Y, Wang Q et al (2019) Chinese clinical named entity recognition using residual dilated convolutional neural network with conditional random field. IEEE Trans Nanobiosci 18(3):306–315. https://doi.org/10.1109/TNB.2019.2908678

    Article  Google Scholar 

  16. Garwood IC, Chakravarty S, Donoghue J et al (2021) A hidden Markov model reliably characterizes ketamine-induced spectral dynamics in macaque local field potentials and human electroencephalograms. PLoS Comput Biol 17(8):e1009280. https://doi.org/10.1371/journal.pcbi.1009280

    Article  Google Scholar 

  17. Tanwar M, Park H, Raghavan N (2021) Multistate diagnosis and prognosis of lubricating oil degradation using sticky hierarchical dirichlet process-hidden Markov model framework. Appl Sci 11(14):6603. https://doi.org/10.3390/app11146603

    Article  Google Scholar 

  18. Cimtay Y, Ekmekcioglu E (2020) Investigating the use of pretrained convolutional neural network on cross-subject and cross-dataset EEG emotion recognition. Sensors 20(7):2034. https://doi.org/10.3390/s20072034

    Article  Google Scholar 

  19. Lee HJ, Lee D (2020) Study of process-focused assessment using an algorithm for facial expression recognition based on a deep neural network model. Electronics 10(1):54. https://doi.org/10.3390/electronics10010054

    Article  Google Scholar 

  20. Lim JZ, Mountstephens J, Teo J (2020) Emotion recognition using eye-tracking: taxonomy, review and current challenges. Sensors 20(8):2384. https://doi.org/10.3390/s20082384

    Article  Google Scholar 

  21. Tan C, Ceballos G, Kasabov N et al (2020) Fusionsense: Emotion classification using feature fusion of multimodal data and deep learning in a brain-inspired spiking neural network. Sensors 20(18):5328. https://doi.org/10.3390/s20185328

    Article  Google Scholar 

  22. Li C, Wang J, Wang H et al (2019) Visual-texual emotion analysis with deep coupled video and danmu neural networks. IEEE Trans Multimed 22(6):1634–1646. https://doi.org/10.1109/TMM.2019.2946477

    Article  Google Scholar 

  23. Nawaz R, Cheah KH, Nisar H et al (2020) Comparison of different feature extraction methods for EEG-based emotion recognition. Biocybern Biomed Eng 40(3):910–926. https://doi.org/10.1016/j.bbe.2020.04.005

    Article  Google Scholar 

  24. Zhu YX, Jin HR (2021) Speaker localization based on audio-visual bimodal fusion. J Adv Comput Intell Intell Inform 25(3):375–382. https://doi.org/10.20965/jaciii.2021.p0375

    Article  Google Scholar 

  25. Ramírez J, Flores MJ (2020) Machine learning for music genre: multifaceted review and experimentation with audioset. J Intell Inf Syst 55(3):469–499

    Article  Google Scholar 

  26. Abdillah J, Asror I, Wibowo YFA (2020) Emotion classification of song lyrics using bidirectional LSTM method with glove word representation weighting. Jurnal RESTI (Rekayasa Sistem Dan Teknologi Informasi) 4(4):723–729. https://doi.org/10.29207/resti.v4i4.2156

    Article  Google Scholar 

  27. Fraiwan L, Hassanin O, Fraiwan M et al (2021) Automatic identification of respiratory diseases from stethoscopic lung sound signals using ensemble classifiers. Biocybern Biomed Eng 41(1):1–14. https://doi.org/10.1016/j.bbe.2020.11.003

    Article  Google Scholar 

  28. Jiang D, Jin D, Zhuang J et al (2021) A computational model of emotion based on audio-visual stimuli understanding and personalized regulation with concurrency. Concurr Comput Pract Exp 33(17):e6269. https://doi.org/10.1002/cpe.6269

    Article  Google Scholar 

  29. Khataei Maragheh H, Gharehchopogh FS, Majidzadeh K et al (2022) A new hybrid based on long Short-term memory network with spotted Hyena optimization algorithm for multi-label text classification. Mathematics 10(3):488. https://doi.org/10.3390/math10030488

    Article  Google Scholar 

  30. Zhou T, Thung KH, Zhu X et al (2019) Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis. Hum Brain Mapp 40(3):1001–1016. https://doi.org/10.1002/hbm.24428

    Article  Google Scholar 

  31. Pham BT, Phong TV, Nguyen HD et al (2020) A comparative study of kernel logistic regression, radial basis function classifier, multinomial naïve bayes, and logistic model tree for flash flood susceptibility mapping. Water 12(1):239. https://doi.org/10.3390/w12010239

    Article  Google Scholar 

  32. Hammami I, Salhi L, Labidi S (2020) Voice pathologies classification and detection using EMD-DWT analysis based on higher order statistic features. Irbm 41(3):161–171. https://doi.org/10.1016/j.irbm.2019.11.004

    Article  Google Scholar 

  33. Suhaimi NS, Mountstephens J, Teo J (2022) A dataset for emotion recognition using virtual reality and EEG (DER-VREEG): emotional state classification using low-cost wearable VR-EEG headsets. Big Data Cogn Comput 6(1):16. https://doi.org/10.3390/bdcc6010016

    Article  Google Scholar 

  34. Liu SH, Wang JJ, Chen W et al (2020) Classification of photoplethysmographic signal quality with fuzzy neural network for improvement of stroke volume measurement. Appl Sci 10(4):1476. https://doi.org/10.3390/app10041476

    Article  Google Scholar 

  35. Xu F, Pan Z, Xia R (2020) E-commerce product review sentiment classification based on a naïve Bayes continuous learning framework. Inf Process Manag 57(5):102221. https://doi.org/10.1016/j.ipm.2020.102221

    Article  Google Scholar 

Download references

Acknowledgments

The authors acknowledge the help from the university colleagues.

Funding

This research received no external funding.

Author information

Authors and Affiliations

  1. School of Music and Dance, Huaihua University, Huaihua, 418000, China

    Chunqiu Wang

  2. School of Education Department, Sehan University, Chonnam, 58447, Korea

    Chunqiu Wang

  3. Department of Teaching Profession, Sehan University, Chonnam, 58447, Korea

    Young Chun Ko

Authors
  1. Chunqiu Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Young Chun Ko
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Young Chun Ko.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, C., Ko, Y.C. Emotional representation of music in multi-source data by the Internet of Things and deep learning. J Supercomput 79, 349–366 (2023). https://doi.org/10.1007/s11227-022-04665-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04665-3

Keywords