Abstract
Intuitive second screen experiences involving smartphone or tablet use are currently not possible with existing VR systems. Whenever users are immersed in a virtual reality world, interactive features and content are usually presented as pop-up windows and floating-text notifications, immediately breaking user immersion, serving as a reminder they are navigating a computer-generated environment. In this paper we discuss how to introduce second screen experiences in VR by orchestrating interplay between using handheld devices while using a VR headset. This entails allowing the users to interface with the touchscreen display despite the opaque, view-obstructing visor. We propose achieving this through a “mobile second-screen interreality system” (MSIS), referring a virtual 3D smart device representation inside the VR world, which is tightly coupled with its real-world counterpart, allowing users to manipulate, interface with, and look at the actual device screen, as if the head-mounted display were not present. This can be achieved via advanced screen-sharing technology, and pixel-perfect mapping between the real device and its virtual “double”, ensuring basic functions (such as typing on an on-screen keyboard and/or tapping on application icons) can be carried out effortlessly. Advanced hand and finger-tracking can be used to represent the user’s virtual hands’ and fingers’ movements on the screen, further enhancing virtual presence, and cementing the links established between the human brain and virtual reality space.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Baratto, R.A., Kim, L.N., Nieh, J.: THINC: a virtual display architecture for thin-client computing. In: ACM SIGOPS Operating Systems Review, vol. 39, no. 5, pp. 277–290. ACM (2005)
Bezegová, E., Ledgard, M. A., Molemaker, R-J., Oberč, B. P., Vigkos, A.: Virtual Reality and its potential for Europe. Ecorys report (2017)
Chandra, S., Boreczky, J., Rowe, L.A.: High performance many-to-many intranet screen sharing with DisplayCast. ACM Trans. Multimedia Comput. Commun. Appl. (TOMM) 10(2), 19 (2014)
Crivellaro, A., Rad, M., Verdie, Y., Moo Yi, K., Fua, P., Lepetit, V.: A novel representation of parts for accurate 3D object detection and tracking in monocular images. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4391–4399 (2015)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893. IEEE (2005)
Einicke, G.A., White, L.B.: Robust extended Kalman filtering. IEEE Trans. Signal Process. 47(9), 2596–2599 (1999)
French, A.M., Guo, C., Shim, J.P.: Current status, issues, and future of bring your own device (BYOD). Commun. Assoc. Inf. Syst. 35(1), 10 (2014)
Gintautas, V., Hübler, A.W.: Experimental evidence for mixed reality states in an interreality system. Phys. Rev. E 75(5), 057201 (2007)
Ko, H.Y., Lee, J.H., Kim, J.O.: Implementation and evaluation of fast mobile VNC systems. IEEE Trans. Consum. Electron. 58(4), 1211–1218 (2012)
Li, X., An, D.: An in-depth performance analysis and optimization for android screencast. In: Proceedings of the 2nd International Conference on Computer Science and Application Engineering, p. 69. ACM (2018)
Low, K.L.: Linear least-squares optimization for point-to-plane ICP surface registration. Chapel Hill Univ. N. C. 4(10), 1–3 (2004)
Margetis, G., Ntoa, S., Stephanidis, C.: Smart omni-channel consumer engagement in malls. In: Stephanidis, C. (ed.) HCII 2019. CCIS, vol. 1034, pp. 89–96. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23525-3_12
Rad, M., Lepetit, V.: BB8: a scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth. In: International Conference on Computer Vision, vol. 1, no. 4, p. 5 (2017)
Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proceedings of the 3rd International Conference on 3-D Digital Imaging and Modeling, pp. 145–152. IEEE (2001)
Sminchisescu, C.: Three-Dimensional Human Modeling and Motion. Reconstruction in Monocular Video Sequences. Doctoral dissertation, Ph. D. Dissertation (2002)
Tabuenca, B., Kalz, M., Löhr, A.: MoocCast: evaluating mobile-screencast for online courses. Univ. Access Inf. Soc. 17(4), 745–753 (2017). https://doi.org/10.1007/s10209-017-0528-x
Wohlhart, P., Lepetit, V.: Learning descriptors for object recognition and 3D pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3109–3118. IEEE (2015)
Yamada, W., Manabe, H., Ikeda, D., Rekimoto, J.: VARiable HMD: optical see-through HMD for AR and VR. In: The Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology, pp. 131–133. ACM (2019)
Yang, S.J., Nieh, J., Selsky, M., Tiwari, N.: The performance of remote display mechanisms for thin-client computing. In: USENIX Annual Technical Conference, General Track, pp. 131–146 (2002)
Zabulis, X., Lourakis, M.I.A., Koutlemanis, P.: Correspondence-free pose estimation for 3D objects from noisy depth data. Vis. Comput. 34(2), 193–211 (2016). https://doi.org/10.1007/s00371-016-1326-9
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Apostolakis, K.C., Margetis, G., Stephanidis, C. (2020). ‘Bring Your Own Device’ in VR: Intuitive Second-Screen Experiences in VR Isolation. In: Stephanidis, C., Antona, M., Ntoa, S. (eds) HCI International 2020 – Late Breaking Posters. HCII 2020. Communications in Computer and Information Science, vol 1294. Springer, Cham. https://doi.org/10.1007/978-3-030-60703-6_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-60703-6_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60702-9
Online ISBN: 978-3-030-60703-6
eBook Packages: Computer ScienceComputer Science (R0)