Abstract
Augmented reality (AR) interaction methods are leaning towards more natural techniques, such as voice commands, hand gestures, and in-air tapping for input. From a security perspective, however, recent works have demonstrated that these methods, such as in-air tapping, are vulnerable to inference attacks where an adversary is capable of reconstructing input in the virtual environment using low-level hand-tracking data with high accuracy. This paper addresses the defense of in-air tapping mechanisms against inference attacks by developing and evaluating a 3D curved keyboard for input. Our design exploits the symmetry between the virtual and physical worlds enabling the inference attack in the first place and increasing the uncertainty of the adversary by manipulating the geometric aspects of this keyboard plane in 3D. We evaluate our design through numerous experiments and show it to be robust against inference attacks, where the adversary’s accuracy in obtaining the correct input text is reduced to 0% (from 87%) and at most to just 18% within the top-500 candidate reconstructions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
High resolution time level 2. http://web.archive.org/web/2017 1017013909/ (2017). Accessed 17 Oct 2017
9241-11:2018, I.: Ergonomics of human-system interaction. https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
Ali, K., Liu, A.X., Wang, W., Shahzad, M.: Keystroke recognition using WiFi signals. In: ACM MobiCom (2015)
Anderson, R., Kuhn, M.: Low cost countermeasures against compromising electromagnetic computer emanations. US Patent 6,721,423 (2004)
Askarov, A., Zhang, D., Myers, A.C.: Predictive black-box mitigation of timing channels. In: ACM CCS (2010)
Asonov, D., Agrawal, R.: Keyboard acoustic emanations. In: IEEE S &P, pp. 3–11 (2004)
Boak, D.G.: A history of us communications security, NSA 1973 (1973)
Brooke, J.: SUS-A Quick and Dirty Usability Scale. CRC Press, Boca Raton (1996)
Chen, S., Wang, J., Guerra, S., Mittal, N., Prakkamakul, S.: Exploring word-gesture text entry techniques in virtual reality. In: CHI EA, pp. 1–6 (2019)
Chizeck, H.J., Bonaci, T.: Brain-computer interface anonymizer. US Patent App. 14/174,818 (2014)
Gupta, A., Ji, C., Yeo, H.S., Quigley, A., Vogel, D.: RotoSwype: word-gesture typing using a ring. In: ACM CHI (2019)
Jin, K., et al.: ViViSnoop: someone is snooping your typing without seeing it! In: IEEE CNS (2017)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) ICLR (2015). http://arxiv.org/abs/1412.6980
Klimt, B., Yang, Y.: The Enron corpus: a new dataset for email classification research. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 217–226. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30115-8_22
Kristensson, P.O., Zhai, S.: Shark2:a large vocabulary shorthand writing system for pen-based computers. In: ACM UIST (2004)
Kuhn, M.G., Anderson, R.J.: Soft tempest: hidden data transmission using electromagnetic emanations. In: Aucsmith, D. (ed.) IH 1998. LNCS, vol. 1525, pp. 124–142. Springer, Heidelberg (1998). https://doi.org/10.1007/3-540-49380-8_10
Liu, X., Zhou, Z., Diao, W., Li, Z., Zhang, K.: When good becomes evil: keystroke inference with smartwatch. In: ACM CCS (2015)
Markussen, A., Jakobsen, M.R., Hornbæk, K.: Vulture: a mid-air word-gesture keyboard. In: ACM CHI (2014)
Meteriz-Yidiran, U., Yildiran, N.F., Awad, A., Mohaisen, D.: A keylogging inference attack on air-tapping keyboards in virtual environments. In: IEEE VR, pp. 765–774 (2022)
Nefes: Data kit untethers USB devices for wireless VR setups. https://bit.ly/3VbqQG8 (2017). Accessed 12 Mar 2020
Shi, C., et al.: Face-mic: Inferring live speech and speaker identity via subtle facial dynamics captured by AR/VR motion sensors. In: ACM MobiCom (2021)
Sun, J., Jin, X., Chen, Y., Zhang, J., Zhang, R., Zhang, Y.: Visible: video-assisted keystroke inference from tablet backside motion. In: NDSS (2016)
Sun, K., Wang, W., Liu, A.X., Dai, H.: Depth aware finger tapping on virtual displays. In: ACM MobiSys (2018)
Tracking: Leap motion controller (2021). https://www.ultraleap.com/product/leap-motion-controller/. Accessed 07 May 2021
Vila, P., Kopf, B.: Loophole: timing attacks on shared event loops in chrome. In: USENIX Security (2017)
Wang, H., Lai, T.T.T., Roy Choudhury, R.: Mole: motion leaks through smartwatch sensors. In: ACM MobiCom (2015)
Yanagihara, N., Shizuki, B.: Cubic keyboard for virtual reality. In: ACM SUI (2018)
Yanagihara, N., Shizuki, B., Takahashi, S.: Text entry method for immersive virtual environments using curved keyboard. In: ACM VRST (2019)
Yildiran, N.F., Meteriz-Yildiran, U., Mohaisen, D.: AiRType: an air-tapping keyboard for augmented reality environments. In: IEEE VR (2022)
Zhang, K., Wang, X.: Peeping tom in the neighborhood: keystroke eavesdropping on multi-user systems. In: USENIX Security (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Althebeiti, H., Gedawy, R., Alghuried, A., Nyang, D., Mohaisen, D. (2024). Defending AirType Against Inference Attacks Using 3D In-Air Keyboard Layouts: Design and Evaluation. In: Kim, H., Youn, J. (eds) Information Security Applications. WISA 2023. Lecture Notes in Computer Science, vol 14402. Springer, Singapore. https://doi.org/10.1007/978-981-99-8024-6_13
Download citation
DOI: https://doi.org/10.1007/978-981-99-8024-6_13
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8023-9
Online ISBN: 978-981-99-8024-6
eBook Packages: Computer ScienceComputer Science (R0)