Akhmajon Makhsadov
André Zenner
ABSTRACT
As immersive virtual experiences find their way into our living room entertainment, they are becoming part of our daily technological consumption. However, state-of-the-art virtual reality (VR) remains disconnected from other digital devices in our environment, such as smartphones or tablets. As context switches between acting in the virtual environment and resolving external notifications negatively influence immersion, we look towards integrating smart devices into virtual experiences. To this aim, we present the VRySmart framework. Through either optical marker tracking or simultaneous localization and mapping (SLAM), embedded smart devices can be used as VR controllers with different levels of integration while their content is incorporated into the virtual context to support the plausibility of the illusion. To investigate user impressions, we conducted a study (N = 10) where participants used a smartphone in four different virtual scenarios. Participants positively assessed smart device usage in VR. We conclude by framing implications for future work.
Supplemental Material
- Jodi Aronson. 1995. A Pragmatic View of Thematic Analysis. The Qualitative Report (April 1995). https://doi.org/10.46743/2160-3715/1995.2069Google Scholar
- Sabah Boustila, Thomas Guégan, Kazuki Takashima, and Yoshifumi Kitamura. 2019. Text Typing in VR Using Smartphones Touchscreen and HMD. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 860–861. https://doi.org/10.1109/VR.2019.8798238Google Scholar
- Florian Daiber, Donald Degraen, André Zenner, Frank Steinicke, Oscar Javier Ariza Núñez, and Adalberto L. Simeone. 2020. Everyday Proxy Objects for Virtual Reality. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3375165Google ScholarDigital Library
- Donald Degraen, Bruno Fruchard, Frederik Smolders, Emmanouil Potetsianakis, Seref Güngör, Antonio Krüger, and Jürgen Steimle. 2021. Weirding Haptics: In-Situ Prototyping of Vibrotactile Feedback in Virtual Reality through Vocalization. Association for Computing Machinery, New York, NY, USA, 936–953. https://doi.org/10.1145/3472749.3474797Google Scholar
- Donald Degraen, Michal Piovarči, Bernd Bickel, and Antonio Krüger. 2021. Capturing Tactile Properties of Real Surfaces for Haptic Reproduction. Association for Computing Machinery, New York, NY, USA, 954–971. https://doi.org/10.1145/3472749.3474798Google Scholar
- Donald Degraen, Anna Reindl, Akhmajon Makhsadov, André Zenner, and Antonio Krüger. 2020. Envisioning Haptic Design for Immersive Virtual Environments. In Companion Publication of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS’ 20 Companion). Association for Computing Machinery, New York, NY, USA, 287–291. https://doi.org/10.1145/3393914.3395870Google ScholarDigital Library
- Donald Degraen, André Zenner, and Antonio Krüger. 2019. Enhancing Texture Perception in Virtual Reality Using 3D-Printed Hair Structures. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300479Google ScholarDigital Library
- Paulo Dias, Luis Afonso, Sérgio Eliseu, and Beatriz Sousa Santos. 2018. Mobile Devices for Interaction in Immersive Virtual Environments. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces (Castiglione della Pescaia, Grosseto, Italy) (AVI ’18). Association for Computing Machinery, New York, NY, USA, Article 19, 9 pages. https://doi.org/10.1145/3206505.3206526Google ScholarDigital Library
- Isamu Endo, Kazuki Takashima, Maakito Inoue, Kazuyuki Fujita, Kiyoshi Kiyokawa, and Yoshifumi Kitamura. 2021. A Reconfigurable Mobile Head-Mounted Display Supporting Real World Interactions. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411763.3451765Google Scholar
- Sarthak Ghosh, Lauren Winston, Nishant Panchal, Philippe Kimura-Thollander, Jeff Hotnog, Douglas Cheong, Gabriel Reyes, and Gregory D. Abowd. 2018. NotifiVR: Exploring Interruptions and Notifications in Virtual Reality. IEEE Transactions on Visualization and Computer Graphics 24, 4(2018), 1447–1456. https://doi.org/10.1109/TVCG.2018.2793698Google ScholarDigital Library
- Daniele Giunchi, Stuart James, Donald Degraen, and Anthony Steed. 2019. Mixing Realities for Sketch Retrieval in Virtual Reality. In The 17th International Conference on Virtual-Reality Continuum and Its Applications in Industry (Brisbane, QLD, Australia) (VRCAI ’19). Association for Computing Machinery, New York, NY, USA, Article 50, 2 pages. https://doi.org/10.1145/3359997.3365751Google ScholarDigital Library
- Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2017. ShareVR: Enabling Co-Located Experiences for Virtual Reality between HMD and Non-HMD Users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4021–4033. https://doi.org/10.1145/3025453.3025683Google ScholarDigital Library
- Jan Gugenheimer, Evgeny Stemasov, Harpreet Sareen, and Enrico Rukzio. 2018. FaceDisplay: Towards Asymmetric Multi-User Interaction for Nomadic Virtual Reality. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173628Google ScholarDigital Library
- Keisuke Hattori and Tatsunori Hirai. 2020. Inside-out Tracking Controller for VR/AR HMD Using Image Recognition with Smartphones. In ACM SIGGRAPH 2020 Posters (Virtual Event, USA) (SIGGRAPH ’20). Association for Computing Machinery, New York, NY, USA, Article 23, 2 pages. https://doi.org/10.1145/3388770.3407430Google Scholar
- Teresa Hirzle, Jan Rixen, Jan Gugenheimer, and Enrico Rukzio. 2018. WatchVR: Exploring the Usage of a Smartwatch for Interaction in Mobile Virtual Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3170427.3188629Google ScholarDigital Library
- Brent Edward Insko. 2001. Passive Haptics Significantly Enhances Virtual Environments. Ph. D. Dissertation. University of North Carolina at Chapel Hill, USA. Advisor(s) Frederick P. Brooks Jr. http://www.cs.unc.edu/techreports/01-017.pdfGoogle Scholar
- Daniel Kharlamov, Brandon Woodard, Liudmila Tahai, and Krzysztof Pietroszek. 2016. TickTockRay: Smartwatch-Based 3D Pointing for Smartphone-Based Virtual Reality. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology(Munich, Germany) (VRST ’16). Association for Computing Machinery, New York, NY, USA, 365–366. https://doi.org/10.1145/2993369.2996311Google ScholarDigital Library
- Youngwon R. Kim and Gerard J. Kim. 2017. HoVR-Type: Smartphone as a typing interface in VR using hovering. In 2017 IEEE International Conference on Consumer Electronics (ICCE). 200–203. https://doi.org/10.1109/ICCE.2017.7889285Google Scholar
- Hai-Ning Liang, Yuwei Shi, Feiyu Lu, Jizhou Yang, and Konstantinos Papangelis. 2016. VRMController: An Input Device for Navigation Activities in Virtual Reality Environments. In Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry - Volume 1(Zhuhai, China) (VRCAI ’16). Association for Computing Machinery, New York, NY, USA, 455–460. https://doi.org/10.1145/3013971.3014005Google ScholarDigital Library
- Fabrice Matulic, Aditya Ganeshan, Hiroshi Fujiwara, and Daniel Vogel. 2021. Phonetroller: Visual Representations of Fingers for Precise Touch Input with Mobile Phones in VR. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445583Google Scholar
- Peter Mohr, Markus Tatzgern, Tobias Langlotz, Andreas Lang, Dieter Schmalstieg, and Denis Kalkofen. 2019. TrackCap: Enabling Smartphones for 3D Interaction on Mobile Head-Mounted Displays. Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300815Google Scholar
- Niels Christian Nilsson, André Zenner, and Adalberto L. Simeone. 2021. Propping Up Virtual Reality With Haptic Proxies. IEEE Computer Graphics and Applications 41, 05 (sep 2021), 104–112. https://doi.org/10.1109/MCG.2021.3097671Google ScholarCross Ref
- Niels Christian Nilsson, André Zenner, Adalberto L. Simeone, Donald Degraen, and Florian Daiber. 2021. Haptic Proxies for Virtual Reality: Success Criteria and Taxonomy. In Proceedings of the 1st Workshop on Everyday Proxy Objects for Virtual Reality. (EPO4VR ’21).Google Scholar
- Rufat Rzayev, Sven Mayer, Christian Krauter, and Niels Henze. 2019. Notification in VR: The Effect of Notification Placement, Task and Environment. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (Barcelona, Spain) (CHI PLAY ’19). Association for Computing Machinery, New York, NY, USA, 199–211. https://doi.org/10.1145/3311350.3347190Google ScholarDigital Library
- Gian-Luca Savino. 2020. Virtual Smartphone: High Fidelity Interaction with Proxy Objects in Virtual Reality. arXiv preprint arXiv:2010.00942(2020).Google Scholar
- Oliver Schneider, Bruno Fruchard, Dennis Wittchen, Bibhushan Raj Joshi, Georg Freitag, Donald Degraen, and Paul Strohmeier. 2022. Sustainable Haptic Design: Improving Collaboration, Sharing, and Reuse in Haptic Design Research. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, 1–5. https://doi.org/10.1145/3491101.3503734Google ScholarDigital Library
- Oliver Schneider, Karon MacLean, Colin Swindells, and Kellogg Booth. 2017. Haptic experience design: What hapticians do and where they need help. International Journal of Human-Computer Studies 107 (2017), 5–21. https://doi.org/10.1016/j.ijhcs.2017.04.004 Multisensory Human-Computer Interaction.Google ScholarDigital Library
- Oliver S. Schneider and Karon E. MacLean. 2014. Improvising design with a Haptic Instrument. In 2014 IEEE Haptics Symposium (HAPTICS). 327–332. https://doi.org/10.1109/HAPTICS.2014.6775476Google Scholar
- Marco Speicher, Jan Ehrlich, Vito Gentile, Donald Degraen, Salvatore Sorce, and Antonio Krüger. 2019. Pseudo-Haptic Controls for Mid-Air Finger-Based Menu Interaction. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312927Google ScholarDigital Library
- Anthony Steed and Simon Julier. 2013. Design and implementation of an immersive virtual reality system based on a smartphone platform. In 2013 IEEE Symposium on 3D User Interfaces (3DUI). 43–46. https://doi.org/10.1109/3DUI.2013.6550195Google ScholarCross Ref
- Balasaravanan Thoravi Kumaravel, Cuong Nguyen, Stephen DiVerdi, and Bjoern Hartmann. 2020. TransceiVR: Bridging Asymmetrical Communication Between VR Users and External Collaborators. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 182–195. https://doi.org/10.1145/3379337.3415827Google ScholarDigital Library
- Kashyap Todi, Donald Degraen, Brent Berghmans, Axel Faes, Matthijs Kaminski, and Kris Luyten. 2016. Purpose-Centric Appropriation of Everyday Objects as Game Controllers. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems(San Jose, California, USA) (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 2744–2750. https://doi.org/10.1145/2851581.2892448Google ScholarDigital Library
- Curtis B. Wilkes, Dan Tilden, and Doug A. Bowman. 2012. 3D User Interfaces Using Tracked Multi-touch Mobile Devices. In Joint Virtual Reality Conference of ICAT - EGVE - EuroVR, Ronan Boulic, Carolina Cruz-Neira, Kiyoshi Kiyokawa, and David Roberts (Eds.). The Eurographics Association. https://doi.org/10.2312/EGVE/JVRC12/065-072Google Scholar
- André Zenner, Donald Degraen, and Antonio Krüger. 2019. Addressing Bystander Exclusion in Shared Spaces During Immersive Virtual Experiences. In Proceedings of the 1st Workshop on Challenges Using Head-Mounted Displays in Shared and Social Spaces (socialHMD ’19).Google Scholar
- André Zenner, Marco Speicher, Sören Klingner, Donald Degraen, Florian Daiber, and Antonio Krüger. 2018. Immersive Notification Framework: Adaptive & Plausible Notifications in Virtual Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3170427.3188505Google ScholarDigital Library
- Li Zhang, Huidong Bai, Mark Billinghurst, and Weiping He. 2020. Is This My Phone? Operating a Physical Smartphone in Virtual Reality. In SIGGRAPH Asia 2020 XR(Virtual Event, Republic of Korea) (SA ’20). Association for Computing Machinery, New York, NY, USA, Article 12, 2 pages. https://doi.org/10.1145/3415256.3421499Google Scholar
Recommendations
A Preliminary Study on Full-Body Haptic Stimulation on Modulating Self-motion Perception in Virtual Reality
Augmented Reality, Virtual Reality, and Computer GraphicsAbstractWe introduce a novel experimental system to explore the role of vibrotactile haptic feedback in Virtual Reality (VR) to induce the self-motion illusion. Self-motion (also called vection) has been mostly studied through visual and auditory stimuli ...
Envisioning Haptic Design for Immersive Virtual Environments
DIS' 20 Companion: Companion Publication of the 2020 ACM Designing Interactive Systems ConferenceCurrent techniques for haptics in immersive virtual environments (IVEs) allow users to perceive materials while exploring virtual surfaces. However, these experiences are usually restricted to the properties defined during the design phase of the IVE. ...
SoEs: Attachable Augmented Haptic on Gaming Controller for Immersive Interaction
UIST '16 Adjunct: Adjunct Proceedings of the 29th Annual ACM Symposium on User Interface Software and TechnologyWe present SoEs (Sword of Elements), an attachable augmented haptic device for enhancing gaming controller in the immersive first-person game. Generally, Player can easily receive visual and auditory feedback through head-mounted displays (HMD) and ...
Comments