Skip to main content
SpringerLink
Log in

SACA Net: Cybersickness Assessment of Individual Viewers for VR Content via Graph-Based Symptom Relation Embedding

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12368))

Included in the following conference series:

Abstract

Recently, cybersickness assessment for VR content is required to deal with viewing safety issues. Assessing physical symptoms of individual viewers is challenging but important to provide detailed and personalized guides for viewing safety. In this paper, we propose a novel symptom-aware cybersickness assessment network (SACA Net) that quantifies physical symptom levels for assessing cybersickness of individual viewers. The SACA Net is designed to utilize the relational characteristics of symptoms for complementary effects among relevant symptoms. The proposed network consists of three main parts: a stimulus symptom context guider, a physiological symptom guider, and a symptom relation embedder. The stimulus symptom context guider and the physiological symptom guider extract symptom features from VR content and human physiology, respectively. The symptom relation embedder refines the stimulus-response symptom features to effectively predict cybersickness by embedding relational characteristics with graph formulation. For validation, we utilize two public 360-degree video datasets that contain cybersickness scores and physiological signals. Experimental results show that the proposed method is effective in predicting human cybersickness with physical symptoms. Further, latent relations among symptoms are interpretable by analyzing relational weights in the proposed network.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Methodology for the subjective assessment of the quality of television pictures. ITU-R BT.500-13 (2012)

    Google Scholar 

  2. Subjective methods for the assessment of stereoscopic 3dtv systems. ITU-R BT.2021 (2012)

    Google Scholar 

  3. Allen, J.: Short term spectral analysis, synthesis, and modification by discrete Fourier transform. IEEE Trans. Acoust. Speech Signal Process. 25(3), 235–238 (1977)

    Article  MATH  Google Scholar 

  4. Basak, D., Pal, S., Patranabis, D.C.: Support vector regression. Neural Inf. Process. Lett. Rev. 11(10), 203–224 (2007)

    Google Scholar 

  5. Bruck, S., Watters, P.A.: Estimating cybersickness of simulated motion using the simulator sickness questionnaire (SSQ): a controlled study. In: CIGV, pp. 486–488 (2009)

    Google Scholar 

  6. Buck, L.E., Young, M.K., Bodenheimer, B.: A comparison of distance estimation in HMD-based virtual environments with different HMD-based conditions. ACM Trans. Appl. Percept. (TAP) 15(3), 1–15 (2018)

    Article  Google Scholar 

  7. Carnegie, K., Rhee, T.: Reducing visual discomfort with HMDs using dynamic depth of field. IEEE Comput. Graph. Appl. 35(5), 34–41 (2015)

    Article  Google Scholar 

  8. Chen, Y., Rohrbach, M., Yan, Z., Shuicheng, Y., Feng, J., Kalantidis, Y.: Graph-based global reasoning networks. In: CVPR, pp. 433–442 (2019)

    Google Scholar 

  9. Chuang, S.W., Chuang, C.H., Yu, Y.H., King, J.T., Lin, C.T.: EEG alpha and gamma modulators mediate motion sickness-related spectral responses. Int. J. Neural Syst. 26(02), 1650007 (2016)

    Article  Google Scholar 

  10. Corbillon, X., De Simone, F., Simon, G.: 360-degree video head movement dataset. In: Proceedings of the 8th ACM on Multimedia Systems Conference, pp. 199–204 (2017)

    Google Scholar 

  11. Dennison, M.S., Wisti, A.Z., D’Zmura, M.: Use of physiological signals to predict cybersickness. Displays 44, 42–52 (2016)

    Article  Google Scholar 

  12. Doweck, I., et al.: Alterations in R-R variability associated with experimental motion sickness. J. Auton. Nerv. Syst. 67(1–2), 31–37 (1997)

    Article  Google Scholar 

  13. Egan, D., Brennan, S., Barrett, J., Qiao, Y., Timmerer, C., Murray, N.: An evaluation of heart rate and electrodermal activity as an objective QoE evaluation method for immersive virtual reality environments. In: QoMEX, pp. 1–6 (2016)

    Google Scholar 

  14. Freina, L., Ott, M.: A literature review on immersive virtual reality in education: state of the art and perspectives. In: eLSE, vol. 1, p. 133. “Carol I” National Defence University (2015)

    Google Scholar 

  15. Gallagher, A.G., et al.: Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann. Surg. 241(2), 364 (2005)

    Article  Google Scholar 

  16. Grantcharov, T.P., Kristiansen, V.B., Bendix, J., Bardram, L., Rosenberg, J., Funch-Jensen, P.: Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br. J. Surg. 91(2), 146–150 (2004)

    Article  Google Scholar 

  17. Jeong, D.K., Yoo, S., Jang, Y.: VR sickness measurement with EEG using DNN algorithm. In: VRST, p. 134 (2018)

    Google Scholar 

  18. Kennedy, R.S., Lane, N.E., Berbaum, K.S., Lilienthal, M.G.: Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3(3), 203–220 (1993)

    Article  Google Scholar 

  19. Kim, A.Y., et al.: Automatic detection of major depressive disorder using electrodermal activity. Sci. Rep. 8(1), 1–9 (2018)

    Article  MathSciNet  Google Scholar 

  20. Kim, H.G., Baddar, W.J., Lim, H., Jeong, H., Ro, Y.M.: Measurement of exceptional motion in VR video contents for VR sickness assessment using deep convolutional autoencoder. In: VRST, p. 36 (2017)

    Google Scholar 

  21. Kim, H.G., Lim, H.T., Lee, S., Ro, Y.M.: VRSA net: VR sickness assessment considering exceptional motion for 360 VR video. IEEE Trans. Image Process. 28(4), 1646–1660 (2018)

    Article  MathSciNet  Google Scholar 

  22. Kim, J., Kim, W., Ahn, S., Kim, J., Lee, S.: Virtual reality sickness predictor: analysis of visual-vestibular conflict and VR contents. In: QoMEX, pp. 1–6 (2018)

    Google Scholar 

  23. Kim, J., Kim, W., Oh, H., Lee, S., Lee, S.: A deep cybersickness predictor based on brain signal analysis for virtual reality contents. In: ICCV, pp. 10580–10589 (2019)

    Google Scholar 

  24. Kim, K., Lee, S., Kim, H.G., Park, M., Ro, Y.M.: Deep objective assessment model based on spatio-temporal perception of 360-degree video for VR sickness prediction. In: ICIP, pp. 3192–3196 (2019)

    Google Scholar 

  25. Kim, Y.Y., Kim, H.J., Kim, E.N., Ko, H.D., Kim, H.T.: Characteristic changes in the physiological components of cybersickness. Psychophysiology 42(5), 616–625 (2005)

    Google Scholar 

  26. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)

    Google Scholar 

  27. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: ICLR (2017)

    Google Scholar 

  28. Lee, S., et al.: Physiological fusion net: quantifying individual VR sickness with content stimulus and physiological response. In: ICIP, pp. 440–444 (2019)

    Google Scholar 

  29. Li, Y., Ouyang, W., Zhou, B., Wang, K., Wang, X.: Scene graph generation from objects, phrases and region captions. In: ICCV, pp. 1261–1270 (2017)

    Google Scholar 

  30. Lin, C.T., Chuang, S.W., Chen, Y.C., Ko, L.W., Liang, S.F., Jung, T.P.: EEG effects of motion sickness induced in a dynamic virtual reality environment. In: EMBC, pp. 3872–3875 (2007)

    Google Scholar 

  31. Lin, C.T., Tsai, S.F., Ko, L.W.: EEG-based learning system for online motion sickness level estimation in a dynamic vehicle environment. IEEE Trans. Neural Netw. Learn. Syst. 24(10), 1689–1700 (2013)

    Article  Google Scholar 

  32. Mawalid, M.A., Khoirunnisa, A.Z., Purnomo, M.H., Wibawa, A.D.: Classification of EEG signal for detecting cybersickness through time domain feature extraction using Naïve Bayes. In: CENIM, pp. 29–34 (2018)

    Google Scholar 

  33. Meehan, M., Insko, B., Whitton, M., Brooks, Jr., F.P.: Physiological measures of presence in stressful virtual environments. In: TOG, pp. 645–652 (2002)

    Google Scholar 

  34. Naqvi, S.A.A., Badruddin, N., Malik, A.S., Hazabbah, W., Abdullah, B.: Does 3D produce more symptoms of visually induced motion sickness? In: EMBC, pp. 6405–6408 (2013)

    Google Scholar 

  35. Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In: ICCV, pp. 1520–1528 (2015)

    Google Scholar 

  36. Padmanaban, N., Ruban, T., Sitzmann, V., Norcia, A.M., Wetzstein, G.: Towards a machine-learning approach for sickness prediction in 360 stereoscopic videos. IEEE Trans. Visual. Comput. Graph. 24(4), 1594–1603 (2018)

    Article  Google Scholar 

  37. Pane, E.S., Khoirunnisaa, A.Z., Wibawa, A.D., Purnomo, M.H.: Identifying severity level of cybersickness from EEG signals using CN2 rule induction algorithm. ICIIBMS 3, 170–176 (2018)

    Google Scholar 

  38. Patrao, B., Pedro, S., Menezes, P.: How to deal with motion sickness in virtual reality. Sciences and Technologies of Interaction, 2015 22 nd, pp. 40–46 (2015)

    Google Scholar 

  39. Qi, M., Li, W., Yang, Z., Wang, Y., Luo, J.: Attentive relational networks for mapping images to scene graphs. In: CVPR, pp. 3957–3966 (2019)

    Google Scholar 

  40. Reason, J.T.: Motion sickness adaptation: a neural mismatch model. J. Roy. Soc. Med. 71(11), 819–829 (1978)

    Article  Google Scholar 

  41. Rebenitsch, L., Owen, C.: Review on cybersickness in applications and visual displays. Virtual Reality 20(2), 101–125 (2016). https://doi.org/10.1007/s10055-016-0285-9

    Article  Google Scholar 

  42. Shaffer, F., Ginsberg, J.: An overview of heart rate variability metrics and norms. Front. Publ. Health 5, 258 (2017)

    Article  Google Scholar 

  43. Singla, A., Fremerey, S., Robitza, W., Raake, A.: Measuring and comparing QoE and simulator sickness of omnidirectional videos in different head mounted displays. In: QoMEX, pp. 1–6 (2017)

    Google Scholar 

  44. Tiiro, A.: Effect of visual realism on cybersickness in virtual reality. University of Oulu (2018)

    Google Scholar 

  45. Wagh, K.P., Vasanth, K.: Electroencephalograph (EEG) based emotion recognition system: a review. In: Saini, H.S., Singh, R.K., Patel, V.M., Santhi, K., Ranganayakulu, S.V. (eds.) Innovations in Electronics and Communication Engineering. LNNS, vol. 33, pp. 37–59. Springer, Singapore (2019). https://doi.org/10.1007/978-981-10-8204-7_5

    Chapter  Google Scholar 

  46. Weech, S., Kenny, S., Barnett-Cowan, M.: Presence and cybersickness in virtual reality are negatively related: a review. Front. Psychol. 10, 158 (2019)

    Article  Google Scholar 

  47. Wei, C.S., Ko, L.W., Chuang, S.W., Jung, T.P., Lin, C.T.: EEG-based evaluation system for motion sickness estimation. In: NER, pp. 100–103 (2011)

    Google Scholar 

  48. Wibirama, S., Nugroho, H.A., Hamamoto, K.: Depth gaze and ECG based frequency dynamics during motion sickness in stereoscopic 3D movie. Entertainment Comput. 26, 117–127 (2018)

    Article  Google Scholar 

  49. Woo, S., Kim, D., Cho, D., Kweon, I.S.: Linknet: relational embedding for scene graph. In: NIPS, pp. 560–570 (2018)

    Google Scholar 

  50. Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.c.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: NIPS, pp. 802–810 (2015)

    Google Scholar 

Download references

Acknowledgement

This work was partly supported by IITP grant (No. 2017-0-00780), IITP grant (No. 2017-0-01779), and BK 21 Plus project.

Author information

Authors and Affiliations

  1. Image and Video Systems Lab, School of Electrical Engineering, KAIST, Daejeon, Korea

    Sangmin Lee, Jung Uk Kim, Seongyeop Kim & Yong Man Ro

  2. Signal Processing Lab, Institute of Electrical Engineering, EPFL, Lausanne, Switzerland

    Hak Gu Kim

Authors
  1. Sangmin Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jung Uk Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hak Gu Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Seongyeop Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yong Man Ro
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yong Man Ro .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 308 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lee, S., Kim, J.U., Kim, H.G., Kim, S., Ro, Y.M. (2020). SACA Net: Cybersickness Assessment of Individual Viewers for VR Content via Graph-Based Symptom Relation Embedding. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12368. Springer, Cham. https://doi.org/10.1007/978-3-030-58592-1_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58592-1_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58591-4

  • Online ISBN: 978-3-030-58592-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation