Prediction of motion sickness degree of stereoscopic panoramic videos based on content perception and binocular characteristics
Introduction
With the gradual popularization of immersive Virtual Reality (VR) system, people's pursuit of visual content is not limited to ordinary three-dimensional videos, but stereoscopic panoramic videos with a stronger sense of immersive experience [1]. VR system has revolutionary potential in the interaction with digital content and has been widely used in online games, medical care, education, agriculture and many other fields [2]. However, while providing an immersive viewing environment for the viewers, VR system may also bring motion sickness (MS) [3], [4], [5], also known as VR sickness or screen sickness, which reduces the Quality of Experience (QoE) of the audiences. Existing studies have shown that more than half of healthy adults feel visual discomfort when viewing 3D movies [6], and 80% of them will suffer from motion sickness within 10 minutes after being immersed in VR environment [7]. The common symptoms of motion sickness are nausea, disorientation and visual discomfort [8]. These bad perception experiences limit the development of VR systems to a certain extent.
From the perspective of human perception, the most important cause of motion sickness is the sensory conflict between the visual perception of virtual stimulus and the vestibular perception of actual head movement [9], which is also the most widely accepted theory at present. This theory had been further studied and expanded [10], [11], [12], but ultimately it is all sensory conflict caused by the inconsistency between the visual information seen by human eyes and the information actually felt.
It is pointed out that individual characteristics (susceptibility) are related to motion sickness [1], [13]. So far, there is no complete solution that can effectively eliminate motion sickness in VR systems without external assistance (e.g. vestibular stimulation) while maintaining a good experience for the viewer. In this case, it is of great significance to predict the degree of motion sickness that may occur when viewing stereoscopic panoramic videos, which can let users know in advance the possible reaction caused by viewing videos content. Therefore, different videos can be recommended for different users to adapt to the motion sickness tolerance of VR content viewing, so that the optimal viewing experience can be achieved. This can also give certain guidance to the production of stereoscopic panoramic video content.
Many existing prediction methods of motion sickness had utilized physiological signal data such as Electroencephalogram (EEG), Galvanic Skin Response (GSR), Electrogastrogram (EGG), Heart Rate (HR) and so on to quantify the degree of motion sickness. However, in actual VR applications, it may be difficult to obtain such information in time, additionally, it is inconvenient. Although sensory conflict is a parameter that cannot be directly measured, it is driven by visual stimuli (i.e. visual movement) in the virtual environment [14]. Under this theoretical analysis, sensory conflict can be modeled from the side by extracting features that can reflect visual movement, such as optical flow of videos content [2], [15]. Kim et al. [15] pointed out that the motion sickness prediction model can take user viewing behavior in the scene into account, which can be modeled in the form of saliency detection [16], [17], because visual attention is a limited and effective resource in Human Visual System (HVS).
HVS has many its own characteristics. Firstly, not all visual stimuli evolve into perceived visual information [18], [19]. Secondly, HVS has multi-scale characteristics, and the retina can process visual information of multiple resolutions at the same time [20]. In addition, stereoscopic visual content can bring realistic visual experience to users, but viewing stereoscopic content for a long time will produce a certain degree of visual discomfort. When stereoscopic perception is combined with VR, it can make visual discomfort and motion sickness symptoms worse. However, most of the existing prediction methods for motion sickness are designed for 2D panoramic videos.
In this paper, a motion sickness prediction method based on content perception and binocular characteristics is proposed for stereoscopic panoramic videos, without using the physiological signals. Deep learning-based motion sickness prediction model is used to extract potential features reflecting the severity of motion sickness in the scene to predict the degree of motion sickness when viewing stereoscopic panoramic videos. The contributions of this paper are as follows:
- •
Considering binocular visual perception, a three-stage simulation of binocular visual perception is carried out for the visual information of the left and right views by deep learning network. Firstly, the left and right views are processed simultaneously, representing the simultaneous vision of the first stage. Secondly, the results are fused to obtain the preliminary results of the second stage. Finally, the fusion results are fed back to the left and right viewpoints respectively to simulate the third stage of binocular vision.
- •
Considering the attention mechanism of the HVS and the multi-resolution characteristics of the retina, the salient sequence of the scene content is calculated, and then it is down-sampled for several times. The down-sampled results are respectively used to weight the saliency of multi-scale scene content features in the deep learning network to reduce the influence of visual information in low-attention regions.
- •
In view of the fact that sensory conflicts in VR viewing environment can not be measured in an intuitive way at present, and considering that sensory conflicts are mostly caused by visual motion stimulation in virtual environment, this paper indirectly models sensory conflicts by calculating the motion characteristics of video content.
The remainder of this paper is organized as follows. Section 2 introduces some related works about predicating the degree of motion sickness. Section 3 describes the proposed method in detail. Section 4 presents the established stereoscopic panoramic video database SPVCD. Section 5 gives the experimental results of the proposed method in comparison with other competing methods. The discussion and conclusions are given in Section 6 and Section 7.
Section snippets
Related work
Motion sickness degree prediction methods can be classified into two categories. In this section, we will first introduce the subjective questionnaire and physiological test based methods, as well as the physiological signals and VR content based objective evaluation methods. Then, the relevant binocular vision characteristics are introduced aiming at the motion sickness prediction of stereoscopic panoramic video.
Proposed method
In this paper, a prediction model of motion sickness degree is proposed based on content perception, which considers the binocular visual effect, HVS attention mechanism and motion information. The framework of the model is shown in Fig. 1.
The upper part of Fig. 1 shows the overall process of the proposed motion sickness prediction model, which mainly includes two modules: content analysis based feature extraction, and time pooling. In the content analysis based feature extraction module, three
Stereoscopic panoramic video database SPVCD
We built a stereoscopic panoramic video comfort database, namely SPVCD, which is used as a benchmark to test the performance of motion sickness degree prediction model. We collected a total of 116 stereoscopic panoramic videos from Stanford database [2], Three-dimensional omnidirectional video database of Tianjin University[52] and website [53], [54], [55], [56], and processed them to some extent. All video sequences have a duration of 20 seconds, a frame rate of 30-90fps, and resolution sizes
Experimental results
Experiments have been conducted to verify the performance of the proposed motion sickness degree prediction model, where Pearson linear correlation coefficient (PLCC), Spearman correlation coefficient (SROCC), Kendall correlation coefficient (KROCC) and Root mean square error (RMSE) are used as the performance indexes. PLCC is used to measure the linear correlation between the subjective evaluation value and the prediction value output by objective model. SROCC utilizes the order of subjective
Discussion
The adverse perceptual experience during VR viewing limits the development of VR system to a certain extent. By predicting the degree of motion sickness of VR content, the ideas for solving the problem of motion sickness may be provided, and this can also guide the production of VR content.
In the multi-scale significance weighting module of the proposed method, the feature maps weighted by three different scales are given the same weight during feature aggregation. However, the content that the
Conclusion
Motion sickness prediction models can be used in a variety of applications, such as VR content creation, VR movies, VR games, etc. An effective motion sickness prediction model can help users choose appropriate content to view according to their susceptibility to improve the quality of their experience, thus promoting the development of VR industry. In this paper, a new prediction model for motion sickness in stereoscopic panoramic video based on content perception and binocular characteristics
CRediT authorship contribution statement
Ziang Lu: Methodology, Software, Writing – original draft. Mei Yu: Conceptualization, Funding acquisition, Methodology, Writing – review & editing. Gangyi Jiang: Conceptualization, Funding acquisition, Supervision. Biwei Chi: Data curation, Software. Qifeng Dong: Validation.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 61871247, and in part by the Natural Science Foundation of Zhejiang Province under Grant LY21F010003. It was also sponsored by the K. C. Wong Magna Fund of Ningbo University.
Ziang Lu received his bachelor of engineering degree from Ningbo University, Ningbo, China, in 2020. He is currently pursuing the master's degree in the Faculty of Information Science and Engineering from Ningbo University. His research interests mainly include prediction of motion sickness degree of stereoscopic panoramic videos, deep learning and image quality assessment.
References (57)
Measurement of postural stability before and after immersion in a virtual environment
Appl. Ergon.
(1999)- et al.
Effects of display type and motion control on cybersickness in a virtual bike simulator
Displays
(JAN 2018) - et al.
A theory on visually induced motion sickness
Displays
(2008) - et al.
Virtual reality sickness predictor: analysis of visual-vestibular conflict and VR contents
Visual attention: the past 25 years
Vis. Res.
(2011)- et al.
Virtual reality induced symptoms and effects VRISE: comparison of head mounted display HMD, desktop and projection display systems
Displays
(2008) - et al.
Vection and cybersickness generated by head-and-display motion in the Oculus rift
Displays
(2017) - et al.
Influence of video content type on user's virtual reality sickness perception and physiological response
Future Gener. Comput. Syst.
(2019) - et al.
Asymmetrically compressed stereoscopic 3D videos: quality assessment and rate-distortion performance evaluation
IEEE Trans. Image Process.
(2017) - et al.
Binocular spatial activity and reverse saliency drivenno-reference stereopair quality assessment
Signal Process. Image Commun.
(2017)
Blind 3D image quality assessment based on self-similarity of binocular features
Neurocomputing
Blind stereoscopic 3D image quality assessment via analysis of naturalness, structure, and binocular asymmetry
Signal Process.
Human-assisted motion annotation
Cybersickness and its severity arising from virtual reality content: a comprehensive study
Sensors
Towards a machine-learning approach for sickness prediction in 360° stereoscopic videos
IEEE Trans. Vis. Comput. Graph.
Towards a better understanding of VR sickness: physical symptom prediction for VR contents
Proc. AAAI Conf. Artif. Intell.
VR sickness versus VR presence: a statistical prediction model
IEEE Trans. Image Process.
VRSA Net: VR sickness assessment considering exceptional motion for 360° VR video
IEEE Trans. Image Process.
Are there side effects to watching 3D movies? A prospective crossover observational study on visually induced motion sickness
PLoS ONE
Visual-vestibular and visuovisual cortical interaction: new insights from fMRI and pet
Ann. N.Y. Acad. Sci.
Motion Sickness
Modeling motion sickness
Pol. J. Aviat. Med. Bioeng. Psychol.
Assessing individual VR sickness through deep feature fusion of VR video and physiological response
IEEE Trans. Circuits Syst. Video Technol.
A metric to quantify virtual scene movement for the study of cybersickness: definition, implementation, and verification
Presence, Teleoper. Virtual Environ.
A deep motion sickness predictor induced by visual stimuli in virtual reality
IEEE Trans. Neural Netw. Learn. Syst.
Saliency in VR: how do people explore virtual environments?
IEEE Trans. Vis. Comput. Graph.
Blind light field image quality assessment by analyzing angular-spatial characteristics
Digit. Signal Process.
Front-End Vision and Multi-Scale Image Analysis: Multi-Scale Computer Vision Theory and Applications, Written in Mathematica
Cited by (3)
Spatiotemporal multi-feature evaluation of visually induced motion sickness in virtual reality
2024, Guangxue Jingmi Gongcheng/Optics and Precision EngineeringMulti-source Information Perception and Prediction for Panoramic Videos
2024, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Artificial Intelligence-based Motion Sickness Detection: A Survey
2023, ICECS 2023 - 2023 30th IEEE International Conference on Electronics, Circuits and Systems: Technosapiens for Saving Humanity
Ziang Lu received his bachelor of engineering degree from Ningbo University, Ningbo, China, in 2020. He is currently pursuing the master's degree in the Faculty of Information Science and Engineering from Ningbo University. His research interests mainly include prediction of motion sickness degree of stereoscopic panoramic videos, deep learning and image quality assessment.
Mei Yu received the B.S. and M.S. degrees from the Hangzhou Institute of Electronics Engineering, Hangzhou, China, in 1990 and 1993, respectively, and the Ph.D. degree from Ajou University, Suwon, South Korea, in 2000. She is currently a Professor with the Faculty of Information Science and Engineering, Ningbo University, Ningbo, China. Her research interests mainly include image/video coding and visual perception.
Gangyi Jiang received the M.S. degree from Hangzhou University in 1992, and the Ph.D. degree from Ajou University, South Korea, in 2000. He is currently a Professor with the Faculty of Information Science and Engineering, Ningbo University, China. His research interests mainly include digital video compression and communications, multiview video coding, image processing, computational photography, and visual quality modeling.
Biwei Chi received the B.S. and M.S. degrees from Ningbo University, Ningbo, China, in 2018 and 2021. He is now a teaching assistant in College of Information Engineering, Jinhua Polytechnic, Jinhua, China. His research interests mainly include image quality assessment and image processing.
Qifeng Dong received his bachelor of engineering degree from NingboTech University, Ningbo, China, in 2021. He is currently pursuing the master's degree in the Faculty of Information Science and Engineering from Ningbo University. His research interests mainly include prediction of motion sickness degree of stereoscopic panoramic videos, deep learning and visual perception.