Abstract
As Augmented and Virtual Reality (AR/VR) technologies become nowadays very popular, there is the need for better interaction in these environments. In this paper, a toolset that facilitates the process of creating interaction in AR and VR environments is presented. The toolset is mainly targeted on mobile devices as the majority of AR and VR applications are exported. The toolset is divided into two main parts; the first part presents the components that are utilized inside the Unity 3D game engine, and the second part involves a webservice to handle the hand tracking task. The separation of hand tracking into a different part allows changes of the algorithms without the need to update the entire package. To evaluate the proposed toolset, a scenario of usage was fabricated in which users -with a computer science background- were shared a package and were asked to complete three tasks and evaluate their experience of using the toolset to create applications. Furthermore, the resulting applications were tested in real case scenarios to examine the overall performance and experience.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. IEEE Comput. Graph. Appl. 21(6), 34–47 (2001). https://doi.org/10.1109/38.963459
Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. E77-D, 1321–1329 (1994)
Coomans, M.K.D., Timmermans, H.J.P.: Towards a taxonomy of virtual reality user interfaces. In: Proceedings 1997 IEEE Conference on Information Visualization (Cat. No. 97TB100165), pp. 279–284 (1997). https://doi.org/10.1109/IV.1997.626531
Doolani, S., et al.: A review of extended reality (XR) technologies for manufacturing training. Technologies 8(4), 77 (2020). https://doi.org/10.3390/technologies8040077
Alalwan, N., Cheng, L., Al-Samarraie, H., Yousef, R., Alzahrani, A.I., Sarsam, S.M.: Challenges and prospects of virtual reality and augmented reality utilization among primary school teachers: a developing country perspective. Stud. Educ. Eval. 66, 100876 (2020). https://doi.org/10.1016/j.stueduc.2020.100876
Christinaki, E., Vidakis, N., Triantafyllidis, G.: A novel educational game for teaching emotion identification skills to preschoolers with autism diagnosis. Comput. Sci. Inf. Syst. 11(2), 723–743 (2014)
Ghazwani, Y., Smith, S.: Interaction in augmented reality: challenges to enhance user experience. In: Proceedings of the 2020 4th International Conference on Virtual and Augmented Reality Simulations, pp. 39–44 (2020). https://doi.org/10.1145/3385378.3385384
Che, Y., Qi, Y.: Detection-guided 3D hand tracking for mobile AR applications. In: 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 386–392 (2021). https://doi.org/10.1109/ISMAR52148.2021.00055
Wang, L., et al.: WiTrace: centimeter-level passive gesture tracking using OFDM signals. IEEE Trans. Mob. Comput. 20(4), 1730–1745 (2021). https://doi.org/10.1109/TMC.2019.2961885
Ismail, A.W., Fadzli, F.E., Faizal, M.S.M.: Augmented reality real-time drawing application with a hand gesture on a handheld interface. In: 2021 IEEE 6th International Conference on Computing, Communication and Automation (ICCCA), pp. 418–423 (2021). https://doi.org/10.1109/ICCCA52192.2021.9666439
Microsoft: Azure Kinect DK. https://azure.microsoft.com/en-us/services/kinect-dk/. Accessed 17 Mar 2022
LeapMotion: Ultraleap for Developers. https://developer.leapmotion.com/. Accessed 17 Mar 2022
Sensoryx: VRfree - VRfree® glove - intuitive VR interaction. https://www.sensoryx.com/products/vrfree/. Accessed 17 Mar 2022
Unity: Unity. https://unity.com/. Accessed 11 Mar 2022
Krauß, V., Boden, A., Oppermann, L., Reiners, R.: Current practices, challenges, and design implications for collaborative AR/VR application development (2021). https://doi.org/10.1145/3411764.3445335
MarÃn-Vega, H., Alor-Hernández, G., Colombo-Mendoza, L.O., Bustos-López, M., ZataraÃn-Cabada, R.: ZeusAR: a process and an architecture to automate the development of augmented reality serious games. Multimed. Tools Appl. 81(2), 2901–2935 (2021). https://doi.org/10.1007/s11042-021-11695-1
Mccallum, S., Boletsis, C.: Augmented reality & gesture-based architecture in games for the elderly. Stud. Health Technol. Inform. 189, 139–144 (2013). https://doi.org/10.3233/978-1-61499-268-4-139
Tian, H., Wang, C., Manocha, D., Zhang, X.: Realtime hand-object interaction using learned grasp space for virtual environments. IEEE Trans. Vis. Comput. Graph. 25(8), 2623–2635 (2019). https://doi.org/10.1109/TVCG.2018.2849381
Liu, H., et al.: High-fidelity grasping in virtual reality using a glove-based system. In: Proceedings - IEEE International Conference on Robotics and Automation, pp. 5180–5186, May 2019. https://doi.org/10.1109/ICRA.2019.8794230
Han, S., Kim, J.: A study on immersion of hand interaction for mobile platform virtual reality contents. Symmetry 9(2) (2017). https://doi.org/10.3390/sym9020022
Kyriakou, P., Hermon, S.: Can I touch this? Using natural interaction in a museum augmented reality system. Digit. Appl. Archaeol. Cult. Herit. 12, 1–9 (2019). https://doi.org/10.1016/j.daach.2018.e00088
Zhang, F., et al.: Mediapipe hands: on-device real-time hand tracking. arXiv Preprint arXiv:2006.10214 (2020)
Lugaresi, C., et al.: MediaPipe: a framework for building perception pipelines. arXiv Preprint arXiv:1906.08172 (2019)
Chunduru, V., Roy, M., Dasari Romit, N.S, Chittawadigi, R.G.: Hand tracking in 3D space using mediapipe and PnP method for intuitive control of virtual globe. In: 2021 IEEE 9th Region 10 Humanitarian Technology Conference (R10-HTC), pp. 1–6 (2021). https://doi.org/10.1109/R10-HTC53172.2021.9641587
Ren, Y., et al.: Hand gesture recognition using 802.11ad mmWave sensor in the mobile device. In: 2021 IEEE Wireless Communications and Networking Conference Workshops (WCNCW), pp. 1–6 (2021). https://doi.org/10.1109/WCNCW49093.2021.9419978
Brooke, J.: SUS: a quick and dirty usability. Usability Eval. Ind. 189(3) (1996)
Sauro, J.: Measuring usability with the system usability scale. https://measuringu.com/sus/. Accessed 04 Apr 2022
Tabuenca, B., Kalz, M., Ternier, S., Specht, M.: Mobile authoring of open educational resources for authentic learning scenarios. Univ. Access Inf. Soc. 15(3), 329–343 (2014). https://doi.org/10.1007/s10209-014-0391-y
Maia, L.F., et al.: LAGARTO: a LocAtion based games AuthoRing TOol enhanced with augmented reality features. Entertain. Comput. 22, 3–13 (2017). https://doi.org/10.1016/j.entcom.2017.05.001
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Logothetis, I., Karampidis, K., Vidakis, N., Papadourakis, G. (2022). Hand Interaction Toolset for Augmented Reality Environments. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2022. Lecture Notes in Computer Science, vol 13445. Springer, Cham. https://doi.org/10.1007/978-3-031-15546-8_17
Download citation
DOI: https://doi.org/10.1007/978-3-031-15546-8_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15545-1
Online ISBN: 978-3-031-15546-8
eBook Packages: Computer ScienceComputer Science (R0)