Skip to main content
SpringerLink
Log in

Modelling effects of S3D visual discomfort in human emotional state using data mining techniques

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent years, the rapid development of diverse media has been evident in disparate fields such as consumer electronics, automotive infotainment and healthcare software. There is a need for innovative methods to assess user perceived Quality of Experience (QoE), as a proxy for consumer satisfaction with such systems and services. Users emotional state plays a key role in QoE; thus, it is necessary to consider it in user experience evaluation and the design process of stereoscopic 3D video content. In the present article we introduce the use of a specially designed model based on a feedforward Multilayer Perception Artificial Neural Network as an appropriate Machine Learning technique for the estimation of human emotional state while viewing various categories of stereoscopic 3D video content. The goal is to design an emotional state estimator based on direct psychophysiological measurements. The considered psychophysiological signals include heart rate (HR) calculated from an echocardiogram (ECG), electro-dermal activity (EDA), and brain activity (BA) in EEG signals. Participants watched a series of 3D video contents varying in terms of visual quality, while the mentioned psychophysiological signals were recorded, and self-reported subjectively experienced emotions using a Self-Assessment Manikin (SAM) questionnaire. The obtained results show that it is possible to construct such a highly precise estimator of emotional states.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Bala Sundar V, Devi T, Saravanan N (2012). Development of a data clustering algorithm for predicting heart. International journal of computer applications (0975 – 888) volume 48 - no.7, June 2012, DOI https://doi.org/10.5120/7358-0095

  2. Barreda-Ángeles M, Pépion R, Bosc E, LeCallet P, Pereda-Baños A (2014) Exploring the effects of 3D visual discomfort on viewers' emotions. In: 2014 IEEE international conference on image processing (ICIP), pp 753–757. https://doi.org/10.1109/ICIP.2014.7025151

    Chapter  Google Scholar 

  3. Berne RM, Levy MN (2000) Principals of physiology, 3rd edn. St. Louis, Mosby

    Google Scholar 

  4. Boucsein W (1992). Electrodermal activity. Plenum Press, New York https://doi.org/10.1007/978-1-4757-5093-5

  5. Bradley M, Lang P (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry 25(1):49–59. https://doi.org/10.1016/0005-7916(94)90063-9

    Article  Google Scholar 

  6. Bradley M M, Lang P J (2007). The international affective picture system (IAPS) in the study of emotion and attention. Series in affective science. Handbook of emotion elicitation and assessment, New York, NY, US: Oxford University press. ;pp. 29-46

  7. Cernea D, Olech P S, Ebert A, Kerren A (2011). Controlling in-vehicle systems with a commercial eeg headset: performance and cognitive load. Proceedings of IRTG 1131 – Visualization of Large and Unstructured Data Sets Workshop 2011. pp. 113–122, https://doi.org/10.4230/OASIcs.VLUDS.2011.113

  8. Chung L, Yoo SK, Yoonj P, Namhyun K, Keesam J, Byungchae L (2005) Using neural network to recognize human emotions from heart rate variability and skin resistance. Conference proceedings: annual international conference of the IEEE engineering in medicine and biology society. Conference 5:5523–5525. https://doi.org/10.1109/IEMBS.2005.1615734

    Article  Google Scholar 

  9. Costadopoulos N, Tien D, Islam MZ (2019) Discovering emotional logic rules from physiological data of individuals. In: In Proc. of the 18th international conference on machine learning and cybernetics (ICMLC), Kobe, Japan, pp 468–474

    Google Scholar 

  10. Dabas H, Sethi C, Dua C, Dalawat M, Sethia D (2018) Emotion classification using EEG signals. In: CSAI '18 proceedings of the 2018 2nd international conference on computer science and artificial intelligence, Shenzhen, China, pp 380–384. https://doi.org/10.1145/3297156.3297177

    Chapter  Google Scholar 

  11. Dawson ME, Schell AM, Filion DL (2000) The Electrodermal system. Chapter 8 in handbook of psychophysiology, 2nd ed. Cambridge University Press ISBN 62634X:200–223

    Google Scholar 

  12. Đorđević Čegar D, Ostojić I, Kukolj D, Pokrić M, Dražić M (2013) Multilayer perceptron model for image quality assessment. 21st telecommunications forum TELFOR (TELFOR), Belgrade, Serbia pp. 765-768. https://doi.org/10.1109/TELFOR.2013.6716342

  13. Drachen A, Nacke LE, Yannakakis G, Lee Pedersen A (2010) Correlation between heart rate, Electrodermal activity and player. Proceeding sandbox '10 proceedings of the 5th ACM SIGGRAPH symposium on video games, pp. 49-54. https://doi.org/10.1145/1836135.1836143

  14. Eibe F, Hall M A, and Witten I H (2016). The WEKA Workbench. Online Appendix for Data Mining: Practical Machine Learning Tools and Techniques. The WEKA Workbench. Morgan Kaufmann Publishers Inc. San Francisco, CA, USA, 4th Edition, 2016, eBook ISBN: 9780128043578

  15. Feldman LA (1995) Valence focus and arousal focus: individual differences in the structure of affective experience. J Pers Soc Psychol 69(I):153–166. https://doi.org/10.1037/0022-3514.69.1.153

    Article  Google Scholar 

  16. Greene S, Thapliyal H, Caban-Holt A (2016) A survey of affective computing for stress detection: evaluating technologies in stress detection for better health. IEEE consumer electronics magazine volume: 5, issue: 4, Oct. 2016, print ISSN: 2162-2248. Electronic ISSN 2162-2256:44–56. https://doi.org/10.1109/MCE.2016.2590178

    Article  Google Scholar 

  17. Harmon Jones E, Price TF, Gamble PA (2012) The influence of the active states on cognitive brooding/narrowing: considering the importance of motivational intensity. Soc Personal Psychol Compass 6(4):314–327. https://doi.org/10.1111/j.1751-9004.2012.00432.x

    Article  Google Scholar 

  18. Holzinger A, Bruschi M, Eder W (2013) On interactive data visualization of physiological low-cost-sensor data with focus on mental stress. Multidisciplinary research and practice for information systems, springer lecture notes in computer science, LNCS 8127, Heidelberg, Berlin, springer, pp. 469-480. https://doi.org/10.1007/978-3-642-40511-2_34

  19. Hu H, Zhu Z, Gao Z, Zheng R (2018) Analysis on biosignal characteristics to evaluate road rage of younger drivers: a driving simulator study. IEEE intelligent vehicles symposium (IV), Changshu, pp. 156-161. https://doi.org/10.1109/IVS.2018.8500444

  20. Hyun Suk K, Daesub Y, Hyun SS, Park CH (2018) Predicting the EEG level of a driver based on driving information. IEEE transactions on intelligent transportation systems, pp. 1524-9050. https://doi.org/10.1109/TITS.2018.2848300

  21. Jirayucharoensak S, Pan-Ngum S, Israsena P (2014) EEG-Based Emotion Recognition Using Deep Learning Network with Principal Component Based Covariate Shift Adaptation The Scientific World Journal:627892. https://doi.org/10.1155/2014/627892

  22. Kim A, Chang M, Yeseul C, Sohyeon J & Kyoungmin L (2018). The effect of immersion on emotional responses to film viewing in a virtual environment. 25th IEEE conference on virtual reality and 3D user Interfaces, pp. 601–602. https://doi.org/10.1109/VR.2018.8446046.

  23. Klotzsche F, Mariola A, Hofmann SM, Nikulin VV, Villringer A, Michael G (2018) Using EEG to decode subjective levels of emotional arousal during an immersive VR roller coaster ride. IEEE Conference on Virtual Reality. https://doi.org/10.1109/VR.2018.8446275

  24. Koelstra S, Muehl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2011) DEAP: a database for emotion analysis using physiological signals. In: IEEE transaction on affective computing. Issue on Naturalistic Affect Resources for System Building and Evaluation, Special in press

    Google Scholar 

  25. Kukolj D, Đorđević Čegar D, Okolišan D, Ostojić I, Sandić-Stanković D, Hewage C (2013) 3D image quality estimation (ANN) based on depth/disparity and 2D metrics. IEEE 14th international symposium on computational intelligence and informatics (CINTI), Budapest, Hungary pp. 125-130. https://doi.org/10.1109/CINTI.2013.6705177

  26. Kwon M, Kang J, Lee M (2013) Emotion classification in movie clips based on 3D fuzzy GIST and EEG signal analysis. International Winter Workshop on Brain-Computer Interface, Gangwo, South Korea, pp 67–68. https://doi.org/10.1109/IWW-BCI.2013.6506633

    Book  Google Scholar 

  27. Lang A, Potter R F, Bolls P (2009). Where psychophysiology meets the media: taking the effects out of mass media research. Media effects: advances in theory and research 3rd ed., New York, NY: Routledge, pp. 185-206

  28. Lin C T, Wu R C, Liang S F, Chao, Yu-Jie Chen W H, Jung T P (2005). EEG-based drowsiness estimation for safety driving using independent component analysis. IEEE transactions on circuits and systems I: regular papers, volume: 52, Issue: 12, pp. 2726–2738, DOI https://doi.org/10.1109/TCSI.2005.857555

  29. Lisetti CL, Nasoz F (2004) Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP J Adv Signal Process 2004(929414):1672–1687. https://doi.org/10.1155/S1110865704406192

    Article  Google Scholar 

  30. Mäki T, Kukolj D, Đorđević Čegar D, Varela M (2013) A reduced-reference parametric model for audiovisual quality of IPTV services. Fifth international workshop on quality of multimedia experience (QoMEX), Klagenfurt am Wörthersee, pp. 6-11. https://doi.org/10.1109/QoMEX.2013.6603193

  31. Mandryk RL, Atkins MS (2007) A fuzzy physiological approach for continuously modelling emotion during interaction with play technologies. International Journal of Human-Computer Studies 65(2007):329–347. https://doi.org/10.1016/j.ijhcs.2006.11.011

    Article  Google Scholar 

  32. Matias Kivikangas J, Ekman I, Chanel G, Järvelä S, Cowley B, Salminen M, Henttonen P, Ravaja N (2011) Review on psychophysiological methods in game research. Journal of Gaming & Virtual Worlds 3(3):181–199. https://doi.org/10.1386/jgvw.3.3.181_1

    Article  Google Scholar 

  33. Millan J D R, Mourino J, Franze M, Cincotti F, Varsta M, Heikkonen J, Babiloni F (2002). A local neural classifier for the recognition of EEG patterns associated to mental tasks. IEEE Transactions on Neural Networks Volume: 13, Issue: 3, May 2002, Print ISSN: 1045-9227 Electronic ISSN: 1941-0093, pp 678-686, DOI https://doi.org/10.1109/TNN.2002.1000132

  34. Miranda-Correa J A, Abadi M K, Sebe N, Patras I (2017). AMIGOS: a dataset for affect, personality and mood research on individuals and groups. ArXiv e-prints

    Google Scholar 

  35. Ooi JSK, Ahmad SA, Chong YZ, Ali SHM, Ai G, Wagatsuma H (2016) Driver emotion recognition framework based on electrodermal activity measurements during simulated driving conditions. IEEE conference on biomedical engineering and sciences (IECBES), Kuala Lumpur, pp. 365-369. https://doi.org/10.1109/IECBES.2016.7843475

  36. Raake A, Varela M, Möller S, Pinheiro A, Pereira M, Perkis A (2013). Definition of quality of experience. Qualinet white paper on definitions of quality of experience, Lausanne, Switzerland: European network on quality of experience, Vol 1.2, pp. 6-7, HAL id: HAL-00977812

  37. Ravaja N (2004) Contributions of psychophysiology to media research: review and recommendations. Media Psychol 6(2):193–235. https://doi.org/10.1207/s1532785xmep0602_4

    Article  Google Scholar 

  38. Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39(6):1161–1178. https://doi.org/10.1037/h0077714

    Article  Google Scholar 

  39. Schrauf M, Simon M, Schmidt E, Kincses W (2011). Assessing drivers’ fatigue state under real traffic conditions using EEG alpha spindles. Proceedings of the Sixth International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Vol 6., https://doi.org/10.17077/drivingassessment.1374

  40. Sivagowry S (2014). A Pragmatic Approach of Preprocessing the Data Set for Heart Disease Prediction. International Journal of Innovative Research in Computer and Communication Engineering Vol. 2 Issue 11, 3297, pp. 6457–6465, ISSN 2320-9798

  41. Soleymani M, Asghari-Esfeden S, Fu Y, Pantic M (2016) Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans Affect Comput 7(1):17–28. https://doi.org/10.1109/TAFFC.2015.2436926

    Article  Google Scholar 

  42. Stickel C, Ebner M, Steinbach-Nordmann S, Searle G, Holzinger A (2009) Emotion detection: application of the valence arousal space for rapid biological usability testing to enhance universal access. Stephanidis, Constantine (ed.) universal access in human-computer interaction. Addressing diversity, lecture notes in computer science, LNCS 5614. Berlin, Heidelberg: springer, pp. 615-624. https://doi.org/10.1007/978-3-642-02707-9_70

  43. Stickel C, Ebner M, Holzinger A (2010) The XAOS metric - understanding visual complexity as measure of usability. Lecture notes in computer science, LNCS 6389. Berlin, Heidelberg: springer, pp. 278-290. https://doi.org/10.1007/978-3-642-16607-5_18

  44. Stickel C, Holzinger A, Felfernig A (2011). Measuring emotions: towards rapid and low cost methodologies. ACM conference on recommender systems, Chicago, United States, 23/10/11.

  45. Sugaya M, Watanabe I, Yoshida R, Chen F (2018). Human Emotional State Analysis During Driving Simulation Experiment Using Bio-Emotion Estimation Method. 2018 IEEE 42nd annual computer software and applications conference (COMPSAC), Tokyo, Japan, pp. 589–594, DOI https://doi.org/10.1109/COMPSAC.2018.10301

  46. Sun H, Bi L, Bisheng C, Yinjing G (2015). EEG-based Safety Driving Performance Estimation and Alertness Using Support Vector Machine. International Journal of Security and Its Applications Vol. 9, No. 6, pp. 125–134, DOI https://doi.org/10.14257/ijsia.2015.9.6.13

  47. Swangnetr M, Kaber DB (2013) Emotional state classification in patient–robot interaction using wavelet analysis and statistics-based feature selection. IEEE Transactions On Human-Machine Systems 43(1):63–75. https://doi.org/10.1109/TSMCA.2012.2210408

    Article  Google Scholar 

  48. Thayer JF, Ahs F, Fredrikson M, Sollers JJ III, Wager TD (2012) A meta-analysis of heart rate variability and neuroimaging studies: implications for heart rate variability as a marker of stress and health. Neurosci Biobehav Rev 36:747–756. https://doi.org/10.1016/j.neubiorev.2011.11.009

    Article  Google Scholar 

  49. Wang C, Lo Y, Huang C (2016) Exploration of video art and colour vision: the application of image change in digital interaction. In: International conference on advanced materials for science and engineering (ICAMSE), Tainan, 2016, pp 512–515. https://doi.org/10.1109/ICAMSE.2016.7840185

    Chapter  Google Scholar 

  50. Watson D, Tellegen A (1985) Toward a consensual structure of mood. Psychol Bull 98(2):219–235. https://doi.org/10.1037/0033-2909.98.2.219

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. RT-RK Institute for Computer Based Systems, Novi Sad, Serbia

    Dragana Đorđević Čegar

  2. Eurecat – Technology Centre of Catalonia, Barcelona, Spain

    Miguel Barreda-Ángeles

  3. Faculty of Technical Science, University of Novi Sad, Novi Sad, Serbia

    Dragan Kukolj

  4. IRCCyN - Institut de Recherche en Communications et en Cybernétique de Nantes, Nantes, France

    Patrick Le Callet

Authors
  1. Dragana Đorđević Čegar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miguel Barreda-Ángeles
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dragan Kukolj
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patrick Le Callet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dragana Đorđević Čegar.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Đorđević Čegar, D., Barreda-Ángeles, M., Kukolj, D. et al. Modelling effects of S3D visual discomfort in human emotional state using data mining techniques. Multimed Tools Appl 79, 19803–19829 (2020). https://doi.org/10.1007/s11042-020-08844-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08844-3

Keywords

Search

Navigation