Prediction of motion sickness degree of stereoscopic panoramic videos based on content perception and binocular characteristics

https://doi.org/10.1016/j.dsp.2022.103787Get rights and content

Abstract

The immersive viewing environment of Virtual Reality (VR) system inevitably brings motion sickness to users, and restricts the development of VR system to a certain extent. In order to solve the problem of motion sickness when users view, it is necessary to predicate the motion sickness degree of the visual content. However, the methods based on subjective questionnaires or physiological signals will become cumbersome and infeasible with the growth of demand and supply of VR content. Most of the existing prediction methods for motion sickness are designed for 2D panoramic video. In this paper, a motion sickness prediction method based on content perception and binocular characteristics is proposed for stereoscopic panoramic video. The proposed method mainly consists of two modules: feature extraction based on content analysis and time pooling. The feature extraction module takes into account the attention mechanism of human visual system and the multi-channel characteristics of retina, and simulates the sensory conflict and the three-stage process of binocular stereoscopic perception in virtual environment. The time pooling module takes into account the time memory effect of human eyes. The proposed method has achieved excellent prediction performance on stereoscopic panoramic video comfort database SPVCD, its predication results have a good correlation with the subjective scores, which shows that the proposed method can effectively predict the motion sickness degree of VR content.

Introduction

With the gradual popularization of immersive Virtual Reality (VR) system, people's pursuit of visual content is not limited to ordinary three-dimensional videos, but stereoscopic panoramic videos with a stronger sense of immersive experience [1]. VR system has revolutionary potential in the interaction with digital content and has been widely used in online games, medical care, education, agriculture and many other fields [2]. However, while providing an immersive viewing environment for the viewers, VR system may also bring motion sickness (MS) [3], [4], [5], also known as VR sickness or screen sickness, which reduces the Quality of Experience (QoE) of the audiences. Existing studies have shown that more than half of healthy adults feel visual discomfort when viewing 3D movies [6], and 80% of them will suffer from motion sickness within 10 minutes after being immersed in VR environment [7]. The common symptoms of motion sickness are nausea, disorientation and visual discomfort [8]. These bad perception experiences limit the development of VR systems to a certain extent.

From the perspective of human perception, the most important cause of motion sickness is the sensory conflict between the visual perception of virtual stimulus and the vestibular perception of actual head movement [9], which is also the most widely accepted theory at present. This theory had been further studied and expanded [10], [11], [12], but ultimately it is all sensory conflict caused by the inconsistency between the visual information seen by human eyes and the information actually felt.

It is pointed out that individual characteristics (susceptibility) are related to motion sickness [1], [13]. So far, there is no complete solution that can effectively eliminate motion sickness in VR systems without external assistance (e.g. vestibular stimulation) while maintaining a good experience for the viewer. In this case, it is of great significance to predict the degree of motion sickness that may occur when viewing stereoscopic panoramic videos, which can let users know in advance the possible reaction caused by viewing videos content. Therefore, different videos can be recommended for different users to adapt to the motion sickness tolerance of VR content viewing, so that the optimal viewing experience can be achieved. This can also give certain guidance to the production of stereoscopic panoramic video content.

Many existing prediction methods of motion sickness had utilized physiological signal data such as Electroencephalogram (EEG), Galvanic Skin Response (GSR), Electrogastrogram (EGG), Heart Rate (HR) and so on to quantify the degree of motion sickness. However, in actual VR applications, it may be difficult to obtain such information in time, additionally, it is inconvenient. Although sensory conflict is a parameter that cannot be directly measured, it is driven by visual stimuli (i.e. visual movement) in the virtual environment [14]. Under this theoretical analysis, sensory conflict can be modeled from the side by extracting features that can reflect visual movement, such as optical flow of videos content [2], [15]. Kim et al. [15] pointed out that the motion sickness prediction model can take user viewing behavior in the scene into account, which can be modeled in the form of saliency detection [16], [17], because visual attention is a limited and effective resource in Human Visual System (HVS).

HVS has many its own characteristics. Firstly, not all visual stimuli evolve into perceived visual information [18], [19]. Secondly, HVS has multi-scale characteristics, and the retina can process visual information of multiple resolutions at the same time [20]. In addition, stereoscopic visual content can bring realistic visual experience to users, but viewing stereoscopic content for a long time will produce a certain degree of visual discomfort. When stereoscopic perception is combined with VR, it can make visual discomfort and motion sickness symptoms worse. However, most of the existing prediction methods for motion sickness are designed for 2D panoramic videos.

In this paper, a motion sickness prediction method based on content perception and binocular characteristics is proposed for stereoscopic panoramic videos, without using the physiological signals. Deep learning-based motion sickness prediction model is used to extract potential features reflecting the severity of motion sickness in the scene to predict the degree of motion sickness when viewing stereoscopic panoramic videos. The contributions of this paper are as follows:

  • Considering binocular visual perception, a three-stage simulation of binocular visual perception is carried out for the visual information of the left and right views by deep learning network. Firstly, the left and right views are processed simultaneously, representing the simultaneous vision of the first stage. Secondly, the results are fused to obtain the preliminary results of the second stage. Finally, the fusion results are fed back to the left and right viewpoints respectively to simulate the third stage of binocular vision.

  • Considering the attention mechanism of the HVS and the multi-resolution characteristics of the retina, the salient sequence of the scene content is calculated, and then it is down-sampled for several times. The down-sampled results are respectively used to weight the saliency of multi-scale scene content features in the deep learning network to reduce the influence of visual information in low-attention regions.

  • In view of the fact that sensory conflicts in VR viewing environment can not be measured in an intuitive way at present, and considering that sensory conflicts are mostly caused by visual motion stimulation in virtual environment, this paper indirectly models sensory conflicts by calculating the motion characteristics of video content.

The remainder of this paper is organized as follows. Section 2 introduces some related works about predicating the degree of motion sickness. Section 3 describes the proposed method in detail. Section 4 presents the established stereoscopic panoramic video database SPVCD. Section 5 gives the experimental results of the proposed method in comparison with other competing methods. The discussion and conclusions are given in Section 6 and Section 7.

Section snippets

Related work

Motion sickness degree prediction methods can be classified into two categories. In this section, we will first introduce the subjective questionnaire and physiological test based methods, as well as the physiological signals and VR content based objective evaluation methods. Then, the relevant binocular vision characteristics are introduced aiming at the motion sickness prediction of stereoscopic panoramic video.

Proposed method

In this paper, a prediction model of motion sickness degree is proposed based on content perception, which considers the binocular visual effect, HVS attention mechanism and motion information. The framework of the model is shown in Fig. 1.

The upper part of Fig. 1 shows the overall process of the proposed motion sickness prediction model, which mainly includes two modules: content analysis based feature extraction, and time pooling. In the content analysis based feature extraction module, three

Stereoscopic panoramic video database SPVCD

We built a stereoscopic panoramic video comfort database, namely SPVCD, which is used as a benchmark to test the performance of motion sickness degree prediction model. We collected a total of 116 stereoscopic panoramic videos from Stanford database [2], Three-dimensional omnidirectional video database of Tianjin University[52] and website [53], [54], [55], [56], and processed them to some extent. All video sequences have a duration of 20 seconds, a frame rate of 30-90fps, and resolution sizes

Experimental results

Experiments have been conducted to verify the performance of the proposed motion sickness degree prediction model, where Pearson linear correlation coefficient (PLCC), Spearman correlation coefficient (SROCC), Kendall correlation coefficient (KROCC) and Root mean square error (RMSE) are used as the performance indexes. PLCC is used to measure the linear correlation between the subjective evaluation value and the prediction value output by objective model. SROCC utilizes the order of subjective

Discussion

The adverse perceptual experience during VR viewing limits the development of VR system to a certain extent. By predicting the degree of motion sickness of VR content, the ideas for solving the problem of motion sickness may be provided, and this can also guide the production of VR content.

In the multi-scale significance weighting module of the proposed method, the feature maps weighted by three different scales are given the same weight during feature aggregation. However, the content that the

Conclusion

Motion sickness prediction models can be used in a variety of applications, such as VR content creation, VR movies, VR games, etc. An effective motion sickness prediction model can help users choose appropriate content to view according to their susceptibility to improve the quality of their experience, thus promoting the development of VR industry. In this paper, a new prediction model for motion sickness in stereoscopic panoramic video based on content perception and binocular characteristics

CRediT authorship contribution statement

Ziang Lu: Methodology, Software, Writing – original draft. Mei Yu: Conceptualization, Funding acquisition, Methodology, Writing – review & editing. Gangyi Jiang: Conceptualization, Funding acquisition, Supervision. Biwei Chi: Data curation, Software. Qifeng Dong: Validation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 61871247, and in part by the Natural Science Foundation of Zhejiang Province under Grant LY21F010003. It was also sponsored by the K. C. Wong Magna Fund of Ningbo University.

Ziang Lu received his bachelor of engineering degree from Ningbo University, Ningbo, China, in 2020. He is currently pursuing the master's degree in the Faculty of Information Science and Engineering from Ningbo University. His research interests mainly include prediction of motion sickness degree of stereoscopic panoramic videos, deep learning and image quality assessment.

References (57)

  • W. Zhou et al.

    Blind 3D image quality assessment based on self-similarity of binocular features

    Neurocomputing

    (2017)
  • G. Yue et al.

    Blind stereoscopic 3D image quality assessment via analysis of naturalness, structure, and binocular asymmetry

    Signal Process.

    (2018)
  • C. Liu et al.

    Human-assisted motion annotation

  • H. Oh et al.

    Cybersickness and its severity arising from virtual reality content: a comprehensive study

    Sensors

    (2022)
  • N. Padmanaban et al.

    Towards a machine-learning approach for sickness prediction in 360° stereoscopic videos

    IEEE Trans. Vis. Comput. Graph.

    (2018)
  • H.G. Kim et al.

    Towards a better understanding of VR sickness: physical symptom prediction for VR contents

    Proc. AAAI Conf. Artif. Intell.

    (2021)
  • W. Kim et al.

    VR sickness versus VR presence: a statistical prediction model

    IEEE Trans. Image Process.

    (2021)
  • H.G. Kim et al.

    VRSA Net: VR sickness assessment considering exceptional motion for 360° VR video

    IEEE Trans. Image Process.

    (2019)
  • A.G. Solimini

    Are there side effects to watching 3D movies? A prospective crossover observational study on visually induced motion sickness

    PLoS ONE

    (2013)
  • T. Brandt et al.

    Visual-vestibular and visuovisual cortical interaction: new insights from fMRI and pet

    Ann. N.Y. Acad. Sci.

    (2002)
  • J.T. Reason et al.

    Motion Sickness

    (1975)
  • R. Lewkowicz

    Modeling motion sickness

    Pol. J. Aviat. Med. Bioeng. Psychol.

    (2016)
  • S. Lee et al.

    Assessing individual VR sickness through deep feature fusion of VR video and physiological response

    IEEE Trans. Circuits Syst. Video Technol.

    (2022)
  • R.H.Y. So et al.

    A metric to quantify virtual scene movement for the study of cybersickness: definition, implementation, and verification

    Presence, Teleoper. Virtual Environ.

    (2001)
  • J. Kim et al.

    A deep motion sickness predictor induced by visual stimuli in virtual reality

    IEEE Trans. Neural Netw. Learn. Syst.

    (2022)
  • V. Sitzmann et al.

    Saliency in VR: how do people explore virtual environments?

    IEEE Trans. Vis. Comput. Graph.

    (2018)
  • Y. Cui et al.

    Blind light field image quality assessment by analyzing angular-spatial characteristics

    Digit. Signal Process.

    (2021)
  • B.M.H. Romeny

    Front-End Vision and Multi-Scale Image Analysis: Multi-Scale Computer Vision Theory and Applications, Written in Mathematica

    (2008)
  • Cited by (3)

    Ziang Lu received his bachelor of engineering degree from Ningbo University, Ningbo, China, in 2020. He is currently pursuing the master's degree in the Faculty of Information Science and Engineering from Ningbo University. His research interests mainly include prediction of motion sickness degree of stereoscopic panoramic videos, deep learning and image quality assessment.

    Mei Yu received the B.S. and M.S. degrees from the Hangzhou Institute of Electronics Engineering, Hangzhou, China, in 1990 and 1993, respectively, and the Ph.D. degree from Ajou University, Suwon, South Korea, in 2000. She is currently a Professor with the Faculty of Information Science and Engineering, Ningbo University, Ningbo, China. Her research interests mainly include image/video coding and visual perception.

    Gangyi Jiang received the M.S. degree from Hangzhou University in 1992, and the Ph.D. degree from Ajou University, South Korea, in 2000. He is currently a Professor with the Faculty of Information Science and Engineering, Ningbo University, China. His research interests mainly include digital video compression and communications, multiview video coding, image processing, computational photography, and visual quality modeling.

    Biwei Chi received the B.S. and M.S. degrees from Ningbo University, Ningbo, China, in 2018 and 2021. He is now a teaching assistant in College of Information Engineering, Jinhua Polytechnic, Jinhua, China. His research interests mainly include image quality assessment and image processing.

    Qifeng Dong received his bachelor of engineering degree from NingboTech University, Ningbo, China, in 2021. He is currently pursuing the master's degree in the Faculty of Information Science and Engineering from Ningbo University. His research interests mainly include prediction of motion sickness degree of stereoscopic panoramic videos, deep learning and visual perception.

    View full text