Abstract
A user interaction technique is needed to manipulate the Augmented Reality (AR) content to make AR deliver realism. However, these problems persist, especially when they involve natural user interaction such as real hands and speech. Speech might be the most crucial part, especially in detecting the user’s voice. The user’s voice usually contains noises that come from the surroundings and every people produces different kinds of voice vibrations. Different genders also produce different voice vibrations. Robust real hand gestures are required to ensure the AR can accelerate the natural user interface. By providing users with more than one way to interact with AR, it tends to produce an efficient way of interaction. This chapter discusses the approach involves three main phases. In the first phase, study the interaction metaphor using gesture and speech in AR. After that, the second phase is carried out to perform a test application for the gesture and speech interaction. This chapter presents an appropriate interaction method that accelerates user interaction, it describes the implementation of gesture and speech interaction to the next level to see how AR could accelerate user interaction.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Farahani, N., et al.: Exploring virtual reality technology and the Oculus Rift for the examination of digital pathology slides. J. Pathol. Inform. 7, 22 (2016)
Ratcliffe, J., Soave, F., Bryan-Kinns, N., Tokarchuk, L., Farkhatdinov, I.: Extended Reality (XR) remote research: a survey of drawbacks and opportunities. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–13, May 2021
Tepper, O.M., et al.: Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room. Plast. Reconstr. Surg. 140(5), 1066–1070 (2017)
Nor’a, M.N.A., Ismail, A.W., Aladin, M.Y.F.: Interactive augmented reality pop-up book with natural gesture interaction for handheld. In: Lee, N. (eds.) Encyclopedia of Computer Graphics and Games, pp. 1–10. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-08234-9_365-1
Liu, W.: Natural user interface-next mainstream product user interface. In: 2010 IEEE 11th International Conference on Computer-Aided Industrial Design & Conceptual Design 1, vol. 1, pp. 203–205. IEEE, November 2010
Norman, D.A.: Natural user interfaces are not natural. Interactions 17(3), 6–10 (2010)
Fu, L.P., Landay, J., Nebeling, M., Xu, Y., Zhao, C.: Redefining natural user interface. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–3, April 2018
Chun, J., Lee, S.: A vision-based 3D hand interaction for marker-based AR. Int. J. Multimedia Ubiquitous Eng. 7(3), 51–58 (2012)
Quek, F., et al.: Multimodal human discourse: gesture and speech. ACM Trans. Comput. Hum. Interact. 9(3), 171–193 (2002)
Piumsomboon, T., Altimira, D., Kim, H., Clark, A., Lee, G., Billinghurst, M.: Grasp-Shell vs gesture-speech: a comparison of direct and indirect natural interaction techniques in augmented reality. In: ISMAR 2014 - IEEE International Symposium on Mixed and Augmented Reality - Science and Technology 2014, Proceedings, pp. 73–82 (2014)
Malkawi, A.M., Srinivasan, R.S.: Multimodal human-computer interaction for immersive visualization: integrating speech-gesture recognitions and augmented reality for indoor environments multimodal HCI-based visualization model immersive (2004)
Ismail, A.W., Billinghurst, M., Sunar, M.S., Yusof, C.S.: Designing an augmented reality multimodal interface for 6DOF manipulation techniques. In: Arai, K., Kapoor, S., Bhatia, R. (eds.) IntelliSys 2018. AISC, vol. 868, pp. 309–322. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-01054-6_22
Nor’a, M.N.A., et al.: Fingertips interaction method in handheld augmented reality for 3D manipulation. In: 2020 IEEE 5th International Conference on Computing Communication and Automation (ICCCA), pp. 161–166. IEEE, October 2020
Aladin, M.Y.F., Ismail, A.W., Ismail, N.A., Rahim, M.S.M.: Object selection and scaling using multimodal interaction in mixed reality. In: IOP Conference Series: Materials Science and Engineering, vol. 979, no. 1, p. 012004. IOP Publishing, November 2020
Plopski, A., Hirzle, T., Norouzi, N., Qian, L., Bruder, G., Langlotz, T.: The eye in extended reality: a survey on gaze interaction and eye tracking in head-worn extended reality. ACM Comput. Surv. (CSUR) 55(3), 1–39 (2022)
Acknowledgement
We appreciate ViCubeLab at Universiti Teknologi Malaysia (UTM) for the equipment and technical assistance. This work has been funded by the Ministry of Higher Education under FRGS, Registration Proposal No: FRGS/1/2020/ICT10/UTM/02/1.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ismail, A.W., Aladin, M.Y.F., Halim, N.A.A., Manaf, M.S.A. (2023). Augmented Reality Using Gesture and Speech Accelerates User Interaction. In: Shaw, R.N., Paprzycki, M., Ghosh, A. (eds) Advanced Communication and Intelligent Systems. ICACIS 2022. Communications in Computer and Information Science, vol 1749. Springer, Cham. https://doi.org/10.1007/978-3-031-25088-0_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-25088-0_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25087-3
Online ISBN: 978-3-031-25088-0
eBook Packages: Computer ScienceComputer Science (R0)