Skip to main content
SpringerLink
Log in

A Pervasive Multi-physiological Signal-Based Emotion Classification with Shapelet Transformation and Decision Fusion

  • Conference paper
  • First Online:
Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom 2021)

  • 1021 Accesses

Abstract

Emotion classification is a hot pot at present. Since physiological signals are objective and difficult to hide, physiological signals are commonly used in emotion classification methods. However, traditional emotional classification based on physiological signals faced the following challenges: low accuracy and low interpretability. For this, this paper proposes a pervasive multi-physiological signal-based emotion classification with shapelet transformation and decision fusion (PMSEC). In PMSEC, the shapelet transformation and feature extraction are carried out for ECG, GSR, and RA, respectively. Following by, six sub-classifiers are constructed for different physiological signals. Lastly, decision-level fusion is implemented to obtain final emotion results. Experimental results show that the proposed PMSEC is not only highly competitive but also has a broad application prospect compared with EEG-based classification method.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Cavallo, F., Semeraro, F., Fiorini, L., Magyar, G., Sinčák, P., Dario, P.: Emotion modelling for social robotics applications: a review. J. Bionic Eng. 15(2), 185–203 (2018). https://doi.org/10.1007/s42235-018-0015-y

    Article  Google Scholar 

  2. Tojo, T., Ono, O., Noh, N.B., Yusof, R.: Interactive Tutor Robot for collaborative e-learning system. Electr. Eng. Jap. 203(3), 22–29 (2018). https://doi.org/10.1541/ieejeiss.137.1373

    Article  Google Scholar 

  3. Basiri, M., Schill, F., Lima, P.U., Floreano, D.: Localization of emergency acoustic sources by micro aerial vehicles. J. Field Robot. 35(2), 187–201 (2018). https://doi.org/10.1002/rob.21733

    Article  Google Scholar 

  4. Choe, Y.: Meaning Versus Information, Prediction Versus Memory, and Question Versus Answer (2021). arXiv e-prints. https://arxiv.org/abs/2107.13393

  5. Bagnall, A., Lines, J., Bostrom, A., Large, J., Keogh, E.: The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Min. Knowl. Disc. 31(3), 606–660 (2016). https://doi.org/10.1007/s10618-016-0483-9

    Article  MathSciNet  Google Scholar 

  6. Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1175–1191 (2001). https://doi.org/10.1109/34.954607

    Article  Google Scholar 

  7. Agrafioti, F., Hatzinakos, D., Anderson, A.K.: ECG pattern analysis for emotion detection. IEEE Trans. Affect. Comput. 3(1), 102–115 (2012). https://doi.org/10.1109/T-AFFC.2011.28

    Article  Google Scholar 

  8. Verma, G.K., Tiwary, U.S.: Multimodal fusion framework: a multiresolution approach for emotion classification and recognition from physiological signals. Neuroimage 102, 162–172 (2014). https://doi.org/10.1016/j.neuroimage.2013.11.007

    Article  Google Scholar 

  9. Hsu, Y., Wang, J., Chiang, W., Hung, C.: Automatic ECG-based emotion recognition in music listening. IEEE Trans. Affect. Comput. 11(1), 85–99 (2020). https://doi.org/10.1109/TAFFC.2017.2781732

    Article  Google Scholar 

  10. Ye, L., Keogh, E.: Time series shapelets: a novel technique that allows accurate, interpretable and fast classification. Data Min. Knowl. Disc. 22, 149–182 (2011). https://doi.org/10.1007/s10618-010-0179-5

    Article  MathSciNet  MATH  Google Scholar 

  11. Rakthanmanon, T., Keogh, E.: Fast shapelets: a scalable algorithm for discovering time series shapelets. In: Proceedings of the 2013 SIAM International Conference on Data Mining, pp. 668–676. Society for Industrial and Applied Mathematics, Philadelphia (2013). https://doi.org/10.1137/1.9781611972832.74

  12. Hills, J., Lines, J., Baranauskas, E., Mapp, J., Bagnall, A.: Classification of time series by shapelet transformation. Data Min. Knowl. Disc. 28(4), 851–881 (2013). https://doi.org/10.1007/s10618-013-0322-1

    Article  MathSciNet  MATH  Google Scholar 

  13. Grabocka, J., Schilling, N., Wistuba, M., et al.: Learning time-series shapelets. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 392–401 (2014). https://doi.org/10.1145/2623330.2623613

  14. Aldhanhani, A., Damiani, E., Mizouni, R., et al.: Framework for traffic event detection using Shapelet Transform. Eng. Appl. Artif. Intell. 82, 226–235 (2019). https://doi.org/10.1016/j.engappai.2019.04.002

    Article  Google Scholar 

  15. Albert, Z., Matthias, F., et al.: Heart rhythm analyzed via shapelets distinguishes sleep from awake. Front. Physiol. 10, 1554−1554 (2019)https://doi.org/10.3389/fphys.2019.01554

  16. Arul, M., Kareem, A.: Shapelets for Earthquake Detection (2019). arXiv e-prints. https://arxiv.org/abs/1911.09086

  17. Zhao, C., Wang, T., Liu, S., et al.: A fast time series shapelet discovery algorithm combining selective extraction and subclass clustering. J. Softw. 31(03), 763–777 (2020). https://doi.org/10.13328/j.cnki.jos.005912

  18. Renard, X., et al.: Random-shapelet: an algorithm for fast shapelet discovery. In: 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp. 1–10 (2015). https://doi.org/10.1109/DSAA.2015.7344782

  19. Lin, J., et al.: Experiencing SAX: a novel symbolic representation of time series. Data Min. Knowl. Disc. 15(2), 107–144 (2007). https://doi.org/10.1007/s10618-007-0064-z

    Article  MathSciNet  Google Scholar 

  20. Ji, C., et al.: A fast shapelet discovery algorithm based on important data points. Int. J. Web Serv. Res. 14(2), 67–80 (2017). https://doi.org/10.4018/IJWSR.2017040104

    Article  Google Scholar 

  21. Ji, C., et al.: A fast shapelet selection algorithm for time series classification. Comput. Netw. 148, 231–240 (2019). https://doi.org/10.1016/j.conet.2018.11.031

    Article  Google Scholar 

  22. Goshvarpour, A., Abbasi, A., Goshvarpour, A.: An accurate emotion recognition system using ECG and GSR signals and matching pursuit method. Biomed. J. 40, 355–368 (2017). https://doi.org/10.1016/j.bj.2017.11.001

    Article  Google Scholar 

  23. Pan, J., Tompkins, W.J.: A real-time QRS detection algorithm. IEEE Trans. Biomed. Eng. 32(3), 230–236 (1985). https://doi.org/10.1109/TBME.1985.325532

    Article  Google Scholar 

  24. Shukla, J., et al.: Feature extraction and selection for emotion recognition from electrodermal activity. IEEE Trans. Affect. Comput. 12(4), 857–869 (2019). https://doi.org/10.1109/TAFFC.2019.2901673

    Article  Google Scholar 

  25. Wei, W., et al.: Emotion recognition based on weighted fusion strategy of multichannel physiological signals. Comput. Intell. Neurosci. 1–9, 2018 (2018). https://doi.org/10.1155/2018/5296523

    Article  Google Scholar 

  26. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of ICNN’95 - International Conference on Neural Networks, vol. 4, pp. 1942–1948 (1995). https://doi.org/10.1109/ICNN.1995.488968

  27. Udhaya Kumar, S., Hannah Inbarani, H.: PSO-based feature selection and neighborhood rough set-based classification for BCI multiclass motor imagery task. Neural Comput. Appl. 28(11), 3239–3258 (2016). https://doi.org/10.1007/s00521-016-2236-5

    Article  Google Scholar 

  28. Khamis, H., et al.: QRS detection algorithm for telehealth electrocardiogram recordings. IEEE Trans. Biomed. Eng. 63(7), 1377–1388 (2016). https://doi.org/10.1109/TBME.2016.2549060

    Article  Google Scholar 

  29. Kim, K.H., Bang, S.W., Kim, S.R.: Emotion recognition system using short-term monitoring of physiological signals. Med. Biol. Eng. Comput. 42, 419–427 (2004). https://doi.org/10.1007/BF02344719

    Article  Google Scholar 

  30. Arul, M., Kareem, A.: Data Anomaly Detection for Structural Health Monitoring of Bridges using Shapelet Transform (2020). arXive-prints. https://doi.org/10.13140/RG.2.2.34565.78567

  31. Yan, W., Li, G.: Research on time series classification based on shapelet. Comput. Sci. 046(001), 29–35 (2019). 10.11896%EF%BC%8Fj.issn.1002-137X.2019.01.005

    Google Scholar 

  32. Yin, Z., et al.: Recognition of emotions using multimodal physiological signals and an ensemble deep learning mode. Comput. Methods Programs Biomed. 140, 93–110 (2017). https://doi.org/10.1016/j.cmpb.2016.12.005

    Article  Google Scholar 

  33. Soleymani, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012). https://doi.org/10.1109/T-AFFC.2011.25

    Article  MathSciNet  Google Scholar 

  34. Ebrahimi, Z., et al.: A review on deep learning methods for ECG arrhythmia classification. Expert Syst. Appl. X 7, 100033 (2020). https://doi.org/10.1016/j.eswax.2020.100033

    Article  Google Scholar 

  35. Li, C., Xu, C., Feng, Z.: Analysis of physiological for emotion recognition with IRS model. Neurocomputing 178, 103–111 (2016). https://doi.org/10.1016/j.neucom.2015.07.112

    Article  Google Scholar 

  36. Mimoun, B.H.W., Zied, L.: Emotion classification in arousal valence model using MAHNOB-HCI database. Int. J. Adv. Comput. Sci. Appl. 8(3) (2017). https://doi.org/10.14569/IJACSA.2017.080344

  37. Ferdinando, H., Seppänen, T., Alasaarela, E.: Comparing features from ECG pattern and HRV analysis for emotion recognition system. In: 2016 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), pp. 1–6 (2016). https://doi.org/10.1109/CIBCB.2016.7758108

  38. Maryam, B., Keivan, M., Fardad, F., Nader, J.D.: A new emotion detection algorithm using extracted features of the different time-series generated from ST intervals Poincaré map. Biomed. Signal Process. Control 59, 101902 (2020). https://doi.org/10.1016/j.bspc.2020.101902

    Article  Google Scholar 

  39. Huang, Y., Yang, J., Liu, S., Pan, J.: Combining facial expressions and electroencephalography to enhance emotion recognition. Future Internet 11, 105 (2019). https://doi.org/10.3390/fi11050105

    Article  Google Scholar 

  40. Siddharth, S., Jung, T.-P., Sejnowski, T.J.: Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing (2019). arXiv e-prints. https://doi.org/10.1109/TAFFC.2019.2916015

  41. Zhang, Y., Cheng, C., Zhang, Y.: Multimodal emotion recognition using a hierarchical fusion convolutional neural network. IEEE Access 9, 7943–7951 (2021). https://doi.org/10.1109/ACCESS.2021.3049516

    Article  Google Scholar 

Download references

Acknowledgments

We are grateful for the support of the Natural Science Foundation of Shandong Province (No. ZR2020LZH008, ZR2020QF112, ZR2019MF071).

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Shandong Normal University, Jinan, China

    Shichao Zhang, Xiangwei Zheng, Mingzhe Zhang, Gengyuan Guo & Cun Ji

  2. Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, China

    Shichao Zhang, Xiangwei Zheng, Mingzhe Zhang, Gengyuan Guo & Cun Ji

Authors
  1. Shichao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiangwei Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingzhe Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gengyuan Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cun Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xiangwei Zheng or Cun Ji .

Editor information

Editors and Affiliations

  1. Shanghai University, Shanghai, China

    Honghao Gao

  2. Xi’an Jiaotong-Liverpool University, Suzhou, China

    Xinheng Wang

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (MP4 54706 kb)

Rights and permissions

Reprints and permissions

Copyright information

© 2021 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, S., Zheng, X., Zhang, M., Guo, G., Ji, C. (2021). A Pervasive Multi-physiological Signal-Based Emotion Classification with Shapelet Transformation and Decision Fusion. In: Gao, H., Wang, X. (eds) Collaborative Computing: Networking, Applications and Worksharing. CollaborateCom 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 406. Springer, Cham. https://doi.org/10.1007/978-3-030-92635-9_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-92635-9_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-92634-2

  • Online ISBN: 978-3-030-92635-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation