Skip to main content
SpringerLink
Log in

A hybrid 2D–3D tangible interface combining a smartphone and controller for virtual reality

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

Virtual reality (VR) controllers are widely used for 3D virtual object selection and manipulation in immersive virtual worlds, while touchscreen-based devices like smartphones or tablets provide precise 2D tangible input. However, VR controllers and touchscreens are used separately in most cases. This research physically integrates a VR controller and a smartphone to create a hybrid 2D–3D tangible interface for VR interactions, combining the strength of both devices. The hybrid interface inherits physical buttons, 3D tracking, and spatial input from the VR controller while having tangible feedback, 2D precise input, and content display from the smartphone’s touchscreen. We review the capabilities of VR controllers and smartphones to summarize design principles and then present a design space with nine typical interaction paradigms for the hybrid interface. We developed an interactive prototype and three application modes to demonstrate the combination of individual interaction paradigms in various VR scenarios. We conducted a formal user study through a guided walkthrough to evaluate the usability of the hybrid interface. The results were positive, with participants reporting above-average usability and rating the system as excellent on four out of six user experience questionnaire scales. We also described two use cases to demonstrate the potential of the hybrid interface.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://developer.vuforia.com/.

  2. http://www.hitl.washington.edu/artoolkit/.

  3. http://www.uco.es/investiga/grupos/ava/node/26.

  4. https://www.oculus.com/quest-2/.

  5. https://www.thingiverse.com/thing:4640517.

  6. https://github.com/Genymobile/scrcpy/.

  7. https://obsproject.com/forum/resources/obs-virtualcam.949/.

  8. https://www.intelrealsense.com/depth-camera-sr305/.

  9. https://www.ueq-online.org/.

  10. https://developer.oculus.com/experimental/passthrough-api/.

References

  • Afonso L, Dias P, Ferreira C, Santos BS (2017) Effect of hand-avatar in a selection task using a tablet as input device in an immersive virtual environment. In: 2017 IEEE symposium on 3D user interfaces (3DUI). IEEE, pp 247–248

  • Alaee G, Deasi AP, Pena-Castillo L, Brown E, Meruvia-Pastor O (2018) A user study on augmented virtuality using depth sensing cameras for near-range awareness in immersive VR. In: IEEE VR’s 4th workshop on everyday virtual reality (WEVR 2018), vol 10

  • Babic T, Reiterer H, Haller M (2018) Pocket6: A 6dof controller based on a simple smartphone application. In: Proceedings of the symposium on spatial user interaction, pp 2–10

  • Bai H, Nassani A, Ens B, Billinghurst M (2017) Asymmetric bimanual interaction for mobile virtual reality. In: ICAT-EGVE, pp 83–86

  • Bai H, Zhang L, Yang J, Billinghurst M (2021) Bringing full-featured mobile phone interaction into virtual reality. Comput Graph 97:42–53

    Article  Google Scholar 

  • Bergé L-P, Serrano M, Perelman G, Dubois E (2014) Exploring smartphone-based interaction with overview+detail interfaces on 3d public displays. In: Proceedings of the 16th international conference on human-computer interaction with mobile devices & services. MobileHCI ’14. Association for Computing Machinery, New York, pp 125–134. https://doi.org/10.1145/2628363.2628374

  • Besançon L, Issartel P, Ammi M, Isenberg T (2016) Hybrid tactile/tangible interaction for 3d data exploration. IEEE Trans Visual Comput Graph 23(1):881–890

    Article  Google Scholar 

  • Biener V, Schneider D, Gesslein T, Otte A, Kuth B, Kristensson PO, Ofek E, Pahud M, Grubert J (2020) Breaking the screen: Interaction across touchscreen boundaries in virtual reality for mobile knowledge workers. IEEE Trans Visual Comput Graph 26(12):3490–3502. https://doi.org/10.1109/TVCG.2020.3023567

    Article  Google Scholar 

  • Billinghurst M, Baldis S, Matheson L, Philips M (1997) 3d palette: A virtual reality content creation tool. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology. VRST ’97, pp. 155–156. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/261135.261163

  • Bornik A, Beichel R, Kruijff E, Reitinger B, Schmalstieg D (2006) A hybrid user interface for manipulation of volumetric medical data. In: 3D user interfaces (3DUI’06). IEEE, pp 29–36

  • Boustila S, Guégan T, Takashima K, Kitamura Y (2019) Text typing in VR using smartphones touchscreen and HMD. In: 2019 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, pp 860–861

  • Bozgeyikli E, Raij A, Katkoori S, Dubey R (2016) Point & teleport locomotion technique for virtual reality. In: Proceedings of the 2016 annual symposium on computer-human interaction in play. CHI PLAY ’16. Association for Computing Machinery, New York, pp 205–216. https://doi.org/10.1145/2967934.2968105

  • Bragdon A, DeLine R, Hinckley K, Morris MR (2011) Code space: Touch + air gesture hybrid interactions for supporting developer meetings. In: Proceedings of the ACM international conference on interactive tabletops and surfaces. ITS ’11. Association for Computing Machinery, New York, pp 212–221. https://doi.org/10.1145/2076354.2076393

  • Brooke J et al (1996) Sus-a quick and dirty usability scale. Usability evaluation in industry 189(194):4–7

    Google Scholar 

  • Büschel W, Mitschick A, Meyer T, Dachselt R (2019) Investigating smartphone-based pan and zoom in 3d data spaces in augmented reality. In: Proceedings of the 21st International conference on human-computer interaction with mobile devices and services. MobileHCI ’19. Association for Computing Machinery, New York. https://doi.org/10.1145/3338286.3340113

  • Chang Y-K, Huang J-W, Chen C-H, Chen C-W, Peng J-W, Hu M-C, Yao C-Y, Chu H-K (2018) A lightweight and efficient system for tracking handheld objects in virtual reality. In: Proceedings of the 24th ACM symposium on virtual reality software and technology, pp 1–2

  • Cheng L-P, Chang L, Marwecki S, Baudisch P (2018) Iturk: Turning passive haptics into active haptics by making users reconfigure props in virtual reality. In: Extended abstracts of the 2018 CHI conference on human factors in computing systems. CHI EA ’18, p. 1. Association for Computing Machinery, New York. https://doi.org/10.1145/3170427.3186482

  • Chen S, Wang J, Guerra S, Mittal N, Prakkamakul S (2019) Exploring word-gesture text entry techniques in virtual reality. In: Extended abstracts of the 2019 CHI conference on human factors in computing systems. CHI EA ’19. Association for Computing Machinery, New York, pp 1–6. https://doi.org/10.1145/3290607.3312762

  • Cho I, Wartell Z (2015) Evaluation of a bimanual simultaneous 7D of interaction technique in virtual environments. In: 2015 IEEE Symposium on 3D User Interfaces (3DUI). IEEE, pp 133–136

  • de Haan G, Griffith EJ, Koutek M, Post FH (2006) Hybrid interfaces in ves: Intent and interaction. In: EGVE, pp 109–118. Citeseer

  • Desai AP, Pena-Castillo L, Meruvia-Pastor O (2017) A window to your smartphone: exploring interaction and communication in immersive VR with augmented virtuality. In: 2017 14th conference on computer and robot vision (CRV), pp 217–224. IEEE

  • Dong Z, Piumsomboon T, Zhang J, Clark A, Bai H, Lindeman R (2020) A comparison of surface and motion user-defined gestures for mobile augmented reality. In: Extended abstracts of the 2020 CHI conference on human factors in computing systems, pp 1–8

  • Drey T, Gugenheimer J, Karlbauer J, Milo M, Rukzio E (2020) VRSketchIn: exploring the design space of pen and tablet interaction for 3d sketching in virtual reality. Association for Computing Machinery, New York, pp 1–14. https://doi.org/10.1145/3313831.3376628

  • Gao L, Bai H, Billinghurst M, Lindeman RW (2020) User behaviour analysis of mixed reality remote collaboration with a hybrid view interface. In: 32nd Australian conference on human-computer interaction. OzCHI ’20. Association for Computing Machinery, New York, pp 629–638. https://doi.org/10.1145/3441000.3441038

  • Grubert J, Witzani L, Ofek E, Pahud M, Kranz M, Kristensson PO (2018) Effects of hand representations for typing in virtual reality. In: 2018 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, pp 151–158

  • Gugenheimer J, Dobbelstein D, Winkler C, Haas G, Rukzio E (2016) Facetouch: Enabling touch interaction in display fixed UIS for mobile virtual reality. In: Proceedings of the 29th annual symposium on user interface software and technology. UIST ’16. Association for Computing Machinery, New York, pp 49–60. https://doi.org/10.1145/2984511.2984576

  • Guiard Y (1988) The kinematic chain as a model for human asymmetrical bimanual cooperation. Adv Psychol 55:205–228

    Article  Google Scholar 

  • Hartmann J, Vogel D (2018) An evaluation of mobile phone pointing in spatial augmented reality. Extended Abstracts of the 2018 CHI conference on human factors in computing systems. CHI EA ’18. Association for Computing Machinery, New York, NY, USA, pp 1–6

  • Hoppe M, Knierim P, Kosch T, Funk M, Futami L, Schneegass S, Henze N, Schmidt A, Machulla T (2018) Vrhapticdrones: Providing haptics in virtual reality through quadcopters. In: Proceedings of the 17th international conference on mobile and ubiquitous multimedia. MUM 2018, pp 7–18. Association for Computing Machinery, New York. https://doi.org/10.1145/3282894.3282898

  • Huang Y-J, Liu K-Y, Lee S-S, Yeh I-C (2021) Evaluation of a hybrid of hand gesture and controller inputs in virtual reality. Int J Human-Comput Interact 37(2):169–180

    Article  Google Scholar 

  • Issartel P, Guéniat F, Isenberg T, Ammi M (2017) Analysis of locally coupled 3d manipulation mappings based on mobile device motion. Presence Teleop Virt 26(1):66–95

    Article  Google Scholar 

  • Jetter H-C, Rädle R, Feuchtner T, Anthes C, Friedl J, Klokmose CN (2020) “In VR, Everything is Possible!”: Sketching and simulating spatially-aware interactive spaces in virtual reality, pp 1–16. Association for Computing Machinery, New York. https://doi.org/10.1145/3313831.3376652

  • Kim YR, Kim GJ (2016) Hovr-type: Smartphone as a typing interface in VR using hovering. In: Proceedings of the 22nd ACM conference on virtual reality software and technology. VRST ’16, pp 333–334. Association for Computing Machinery, New York. https://doi.org/10.1145/2993369.2996330

  • Kim MJ, Maher ML (2008) The impact of tangible user interfaces on designers’ spatial cognition. Human-Comput Interact 23(2):101–137

    Article  Google Scholar 

  • Kim H, Park J-H (2014) Duplicatespace: Enhancing operability of virtual 3d objects by asymmetric bimanual interaction. In: 2014 11th international conference on ubiquitous robots and ambient intelligence (URAI). IEEE, pp 324–327

  • Kytö M, Ens B, Piumsomboon T, Lee GA, Billinghurst M (2018) Pinpointing: precise head- and eye-based target selection for augmented reality. Association for Computing Machinery, New York, pp 1–14. https://doi.org/10.1145/3173574.3173655

  • Laugwitz B, Held T, Schrepp M (2008) Construction and evaluation of a user experience questionnaire. In: Symposium of the Austrian HCI and usability engineering group. Springer, pp 63–76

  • LaViola JJ Jr (2000) A discussion of cybersickness in virtual environments. ACM Sigchi Bull 32(1):47–56

    Article  Google Scholar 

  • Le HV, Mayer S, Wolf K, Henze N (2016) Finger placement and hand grasp during smartphone interaction. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. CHI EA ’16. Association for Computing Machinery, New York, pp 2576–2584. https://doi.org/10.1145/2851581.2892462

  • Lee C-J, Chu H-K (2018) Dual-MR: Interaction with mixed reality using smartphones. In: Proceedings of the 24th ACM symposium on virtual reality software and technology. VRST ’18. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3281505.3281618

  • Liang H-N, Shi Y, Lu F, Yang J, Papangelis K (2016) Vrmcontroller: An input device for navigation activities in virtual reality environments. In: Proceedings of the 15th ACM SIGGRAPH conference on virtual-reality continuum and its applications in industry—Volume 1. VRCAI ’16. Association for Computing Machinery, New York, pp 455–460. https://doi.org/10.1145/3013971.3014005

  • Lindeman RW, Sibert JL, Hahn JK (1999) Hand-held windows: towards effective 2D interaction in immersive virtual environments. In: Proceedings IEEE virtual reality (Cat. No. 99CB36316). IEEE, pp 205–212

  • Manuri F, Piumatti G (2015) A preliminary study of a hybrid user interface for augmented reality applications. In: 2015 7th international conference on intelligent technologies for interactive entertainment (INTETAIN). IEEE, pp 37–41

  • Matulic F, Ganeshan A, Fujiwara H, Vogel D (2021) Phonetroller: visual representations of fingers for precise touch input with mobile phones in VR. Association for Computing Machinery, New York. https://doi.org/10.1145/3411764.3445583

  • Ma X, Yao Z, Wang Y, Pei W, Chen H (2018) Combining brain-computer interface and eye tracking for high-speed text entry in virtual reality. In: 23rd international conference on intelligent user interfaces. IUI ’18. Association for Computing Machinery, New York, pp 263–267. https://doi.org/10.1145/3172944.3172988

  • Menzner T, Gesslein T, Otte A, Grubert J (2020) Above surface interaction for multiscale navigation in mobile virtual reality. In: 2020 IEEE conference on virtual reality and 3D user interfaces (VR), pp 372–381. https://doi.org/10.1109/VR46266.2020.00057

  • Millette A, McGuffin MJ (2016) Dualcad: integrating augmented reality with a desktop GUI and smartphone interaction. In: 2016 IEEE international symposium on mixed and augmented reality (ISMAR-Adjunct). IEEE, pp 21–26

  • Mine M, Yoganandan A, Coffey D (2014) Making VR work: Building a real-world immersive modeling application in the virtual world. In: Proceedings of the 2nd ACM symposium on spatial user interaction. SUI ’14. Association for Computing Machinery, New York, pp 80–89. https://doi.org/10.1145/2659766.2659780

  • Mohr P, Tatzgern M, Langlotz T, Lang A, Schmalstieg D, Kalkofen D (2019) TrackCap: enabling smartphones for 3D interaction on mobile head-mounted displays. Association for Computing Machinery, New York, pp 1–11. https://doi.org/10.1145/3290605.3300815

  • Normand E, McGuffin MJ (2018) Enlarging a smartphone with AR to create a handheld VESAD (virtually extended screen-aligned display). In: 2018 IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, pp 123–133

  • Schmidt D, Seifert J, Rukzio E, Gellersen H (2012) A cross-device interaction style for mobiles and surfaces. In: Proceedings of the designing interactive systems conference. DIS ’12. Association for Computing Machinery, New York, pp 318–327. https://doi.org/10.1145/2317956.2318005

  • Schwind V, Knierim P, Tasci C, Franczak P, Haas N, Henze N (2017) “These are not my hands!” effect of gender on the perception of avatar hands in virtual reality. In: Proceedings of the 2017 CHI conference on human factors in computing systems, pp 1577–1582

  • Sharmila A (2021) Hybrid control approaches for hands-free high level human-computer interface-a review. J Med Eng Technol 45(1):6–13

    Article  Google Scholar 

  • Sommer B, Wang SJ, Xu L, Chen M, Schreiber F (2015) Hybrid-dimensional visualization and interaction-integrating 2d and 3d visualization with semi-immersive navigation techniques. In: 2015 Big Data Visual Analytics (BDVA). IEEE, pp 1–8

  • Steed A, Julier S (2013) Design and implementation of an immersive virtual reality system based on a smartphone platform. In: 2013 IEEE symposium on 3D user interfaces (3DUI). IEEE, pp 43–46

  • Surale HB, Gupta A, Hancock M, Vogel D (2019) TabletInVR: Exploring the design space for using a multi-touch tablet in virtual reality. Association for Computing Machinery, New York, pp 1–13. https://doi.org/10.1145/3290605.3300243

  • Takashina T, Ikeya M, Tamura T, Nakazumi M, Nomura T, Kokumai Y (2018) Real-virtual bridge: Operating real smartphones from the virtual world. In: Proceedings of the 2018 ACM international conference on interactive surfaces and spaces. ISS ’18, pp 449–452. Association for Computing Machinery, New York. https://doi.org/10.1145/3279778.3279921

  • Usoh M, Arthur K, Whitton MC, Bastos R, Steed A, Slater M, Brooks Jr FP (1999) Walking\(>\) walking-in-place\(>\) flying, in virtual environments. In: Proceedings of the 26th annual conference on computer graphics and interactive techniques, pp 359–364

  • Vanukuru R, Murugan A, Pillai J (2020) Dual phone AR: Exploring the use of phones as controllers for mobile augmented reality. In: 26th ACM symposium on virtual reality software and technology. VRST ’20. Association for Computing Machinery, New York. https://doi.org/10.1145/3385956.3422113

  • Ventura J, Arth C, Reitmayr G, Schmalstieg D (2014) Global localization from monocular slam on a mobile phone. IEEE Trans Visual Comput Graph 20(4):531–539

    Article  Google Scholar 

  • Vinayak, Ramanujan D, Piya C, Ramani K (2016) Mobisweep: exploring spatial design ideation using a smartphone as a hand-held reference plane. In: Proceedings of the TEI ’16: tenth international conference on tangible, embedded, and embodied interaction. TEI ’16. Association for Computing Machinery, New York, pp 12–20. https://doi.org/10.1145/2839462.2839490

  • Wang S, He W, Zheng B, Feng S, Wang S, Bai X, Billinghurst M (2019) Holding virtual objects using a tablet for tangible 3D sketching in VR. In: 2019 IEEE international symposium on mixed and augmented reality adjunct (ISMAR-Adjunct). IEEE, pp 156–157

  • Wang S, Zhang L, Kang J, Wang S, He W (2021) Vrsmartphonesketch: Augmenting VR controller with a smartphone for mid-air sketching. In: 2021 IEEE international symposium on mixed and augmented reality adjunct (ISMAR-Adjunct), pp 362–363. https://doi.org/10.1109/ISMAR-Adjunct54149.2021.00082

  • Wyeth P (2008) Understanding engagement with tangible user interfaces. In: Proceedings of the 20th Australasian conference on computer-human interaction: designing for habitus and habitat, pp 331–334

  • Zhang L, Bai H, Billinghurst M, He W (2020) Is this my phone? operating a physical smartphone in virtual reality. In: SIGGRAPH Asia 2020 XR. SA ’20. Association for Computing Machinery, New York. https://doi.org/10.1145/3415256.3421499

  • Zhang L, He W, Bai H, He J, Qiao Y, Billinghurst M (2021a) A hybrid 2d-3d tangible interface for virtual reality. In: ACM SIGGRAPH 2021 Posters. SIGGRAPH ’21. Association for Computing Machinery, New York. https://doi.org/10.1145/3450618.3469166

  • Zhang L, He W, Cao Z, Wang S, Bai H, Billinghurst M (2021b) Simultaneous real walking and asymmetric input in virtual reality with a smartphone-based hybrid interface. In: 2021 IEEE international symposium on mixed and augmented reality adjunct (ISMAR-Adjunct). IEEE Computer Society, Los Alamitos, pp 321–323. https://doi.org/10.1109/ISMAR-Adjunct54149.2021.00072

  • Zhang L, He W, Liu Y, Zou Q, Bai H, Billinghurst M (2021c) Arinvr: Bringing mobile AR into VR. In: SIGGRAPH Asia 2021 Posters. SA ’21 Posters. Association for Computing Machinery, New York. https://doi.org/10.1145/3476124.3488636

  • Zhu F, Grossman T (2020) BISHARE: exploring bidirectional interactions between smartphones and head-mounted augmented reality. Association for Computing Machinery, New York, pp 1–14. https://doi.org/10.1145/3313831.3376233

Download references

Funding

This work was partially supported by the National Key R &D Program of China (Grant Nos. 2019YFB1703800, 2021YFB1714900, 2021YFB1716200, 2020YFB1712503), the Programme of Introducing Talents of Discipline to Universities (111 Project), China (Grant No. B13044), the Fundamental Research Funds for the Central Universities, NPU (Grant No. 3102020gxb003).

Author information

Authors and Affiliations

  1. Cyber-Physical Interaction Lab, Northwestern Polytechnical University, 127 West Youyi Road, Xi’an, 710072, China

    Li Zhang, Weiping He & Shuxia Wang

  2. Auckland Bioengineering Institute, The University of Auckland, 70 Symonds St., Auckland, 1010, New Zealand

    Huidong Bai, Qianyuan Zou & Mark Billinghurst

Authors
  1. Li Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weiping He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huidong Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qianyuan Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shuxia Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mark Billinghurst
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LZ was involved in the conceptualization, methodology, validation, software, formal analysis, data Curation, writing—original draft and visualization. WH contributed to the conceptualization, resources, writing—review and editing, supervision and funding acquisition. HB assisted in the conceptualization, methodology, validation, writing—review and editing. QZ helped in writing—original draft and visualization. SW contributed to the resources, writing—review and editing and funding acquisition. MB contributed to the conceptualization, writing—review and editing and supervision.

Corresponding authors

Correspondence to Li Zhang or Weiping He.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethics approval

Approval of all ethical and experimental procedures and protocols was granted by the Medical and Animal Care Ethics Committee, Northwestern Polytechnical University (No. 202202048).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., He, W., Bai, H. et al. A hybrid 2D–3D tangible interface combining a smartphone and controller for virtual reality. Virtual Reality 27, 1273–1291 (2023). https://doi.org/10.1007/s10055-022-00735-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-022-00735-2

Keywords

Search

Navigation