Abstract
Distributed Virtual Reality (DVR) systems enable geographically dispersed users to interact in a shared virtual environment. The realism of the interaction is crucial to increase the feeling of co-presence. Latency, produced either by hard- or software components of DVR applications, impedes reaching high realism levels of the DVR experience. For example, the time delay between the user’s motion and the corresponding display rendering of the DVR system might lead to adverse effects such as a reduced sense of presence or motion sickness. One way of minimizing the latency is to predict user’s motion and thus compensate for the inherent latency in the system. In order to address this problem, we propose a neural network 3D pose tracking and prediction system with latency guarantees for end-to-end avatar reconstruction. We evaluate and compare our system against multiple traditional methods and provide a thorough analysis on real-world human motion data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
An overview on the VIRTOOAIR framework is available at: https://audi-konfuzius-institut-ingolstadt.de/category/akii-microlab/current-projects.
- 2.
Source code available at: https://gitlab.com/akii-microlab/virtooair.
References
Openvr sdk (2015). https://github.com/ValveSoftware/openvr
Vive vr system (2018). https://www.vive.com/us/product/vive-virtual-reality-system/
Allen, B., Curless, B., Popović, Z., Hertzmann, A.: Learning a correlated model of identity and pose-dependent body shape variation for real-time synthesis. In: Proceedings of the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA 2006, pp. 147–156. Eurographics Association, Aire-la-Ville (2006). http://dl.acm.org/citation.cfm?id=1218064.1218084
Becher, A., Angerer, J., Grauschopf, T.: Novel approach to measure motion-to-photon and mouth-to-ear latency in distributed virtual reality systems. In: GI VR/AR WORKSHOP 2018 (2018)
Becher A., Axenie C., Grauschopf, T.: VIRTOOAIR: virtual reality toolbox for avatar intelligent reconstruction. In: Multimodal Virtual and Augmented Reality Workshop (MVAR) at 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (2018)
Choi, S.W., Lee, S., Seo, M.W., Kang, S.J.: Time sequential motion-to-photon latency measurement system for virtual reality head-mounted displays. Electronics 7(9), 171 (2018)
Chollet, F., et al.: Keras (2015). https://github.com/fchollet/keras
Collet, A., et al.: High-quality streamable free-viewpoint video. ACM Trans. Graph. 34(4), 69:1–69:13 (2015). https://doi.org/10.1145/2766945
Daanen, H., Ter Haar, F.: Review. Displays 34(4), 270–275 (2013)
Du, X., Vasudevan, R., Johnson-Roberson, M.: Bio-LSTM: a biomechanically inspired recurrent neural network for 3-d pedestrian pose and gait prediction. IEEE Robot. Autom. Lett. 4(2), 1501–1508 (2019)
Elbamby, M.S., Perfecto, C., Bennis, M., Doppler, K.: Toward low-latency and ultra-reliable virtual reality. IEEE Netw. 32(2), 78–84 (2018)
Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6m: large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1325–1339 (2014)
Jain, A., Thormählen, T., Seidel, H.P., Theobalt, C.: Moviereshape: tracking and reshaping of humans in videos. In: ACM SIGGRAPH Asia 2010 Papers, SIGGRAPH ASIA 2010, pp. 148:1–148:10. ACM, New York (2010). https://doi.org/10.1145/1866158.1866174
Jerald, J., Whitton, M., Brooks Jr., F.P.: Scene-motion thresholds during head yaw for immersive virtual environments. ACM Trans. Appl. Percept. 9(1), 4:1–4:23 (2012)
Lee, K., Lee, I., Lee, S.: Propagating LSTM: 3D pose estimation based on joint interdependency. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 123–141. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_8
Lin, M., Lin, L., Liang, X., Wang, K., Cheng, H.: Recurrent 3D pose sequence machines. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Malleson, C., et al.: Rapid one-shot acquisition of dynamic VR avatars. In: 2017 IEEE Virtual Reality (VR), pp. 131–140 (2017)
Martinez, J., Black, M.J., Romero, J.: On human motion prediction using recurrent neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4674–4683 (2017)
Meehan, M., Razzaque, S., Whitton, M.C., Brooks, F.P.: Effect of latency on presence in stressful virtual environments. In: IEEE Virtual Reality 2003, pp. 141–148, 22–26 March 2003
Munafo, J., Diedrick, M., Stoffregen, T.A.: The virtual reality head-mounted display oculus rift induces motion sickness and is sexist in its effects. Exp. Brain Res. 235(3), 889–901 (2017)
Niehorster, D.C., Li, L., Lappe, M.: The accuracy and precision of position and orientation tracking in the HTC vive virtual reality system for scientific research. i-Perception (2017)
Shih, S., Shih, W.: Application of ridge regression analysis to water resources studies. J. Hydrol. 40(1), 165–174 (1979)
St. Pierre, M.E., Banerjee, S., Hoover, A.W., Muth, E.R.: The effects of 0.2hz varying latency with 20–100ms varying amplitude on simulator sickness in a helmet mounted display. Displays 36, 1–8 (2015)
Tang, Y., Ma, L., Liu, W., Zheng, W.S.: Long-term human motion prediction by modeling motion context and enhancing motion dynamics. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (2018). https://doi.org/10.24963/ijcai.2018/130
Zhou, S., Fu, H., Liu, L., Cohen-Or, D., Han, X.: Parametric reshaping of human bodies in images. ACM Trans. Graph. 29(4), 126:1–126:10 (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Pohl, S., Becher, A., Grauschopf, T., Axenie, C. (2019). Neural Network 3D Body Pose Tracking and Prediction for Motion-to-Photon Latency Compensation in Distributed Virtual Reality. In: Tetko, I., Kůrková, V., Karpov, P., Theis, F. (eds) Artificial Neural Networks and Machine Learning – ICANN 2019: Image Processing. ICANN 2019. Lecture Notes in Computer Science(), vol 11729. Springer, Cham. https://doi.org/10.1007/978-3-030-30508-6_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-30508-6_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-30507-9
Online ISBN: 978-3-030-30508-6
eBook Packages: Computer ScienceComputer Science (R0)