research-article

Modality and Depth in Touchless Smartphone Augmented Reality Interactions

Authors:

David A. Shamma,

Daniel Avrahami,

Jacob BiehlAuthors Info & Claims

IMX '20: Proceedings of the 2020 ACM International Conference on Interactive Media Experiences

Pages 74 - 81

https://doi.org/10.1145/3391614.3393648

Published: 17 June 2020 Publication History

Abstract

Augmented reality (AR) on smartphone devices allows people to interact with virtually placed objects anchored in the real world through the device’s viewport. Typically, smartphone AR interactions work with the device’s 2D touchscreen disconnected from the modality and depth of the virtual objects. In this paper, we studied 15 participants’ preferences, performances, and cognitive loads on a set of common tasks performed on smartphones at two interaction depths (close-range and distant) with two touchless modalities (hand tracking and screen dwell). We find that distant AR interactions, strongly preferred by the participants, were significantly faster and took less cognitive effort. We observed that within both interaction depths, modalities performed equally. When designing touchless modalities on smartphones, we suggest using distant interactions when overall performance is the top priority, otherwise, using hand tracking or screen dwell as back-ups for each other can be equally effective.

References

[1]

Roland Aigner, Daniel Wigdor, Hrvoje Benko, Michael Haller, David Lindbauer, Alexandra Ion, Shengdong Zhao, and JTKV Koh. 2012. Understanding mid-air hand gestures: A study of human preferences in usage of gesture types for hci. Microsoft Research TechReport MSR-TR-2012-111 2 (2012).

[2]

Huidong Bai, Gun A Lee, Mukundan Ramakrishnan, and Mark Billinghurst. 2014. 3D gesture interaction for handheld augmented reality. In SIGGRAPH Asia 2014 Mobile Graphics and Interactive Applications. ACM, ACM, 7.

[3]

Evan Barba and Ramon Zamora Marroquin. 2017. A Primer on Spatial Scale and Its Application to Mixed Reality. In Mixed and Augmented Reality (ISMAR), 2017 IEEE International Symposium on. IEEE, IEEE, 100–110.

[4]

Mark Billinghurst, Tham Piumsomboon, and Huidong Bai. 2014. Hands in space: Gesture interaction with augmented-reality interfaces. IEEE computer graphics and applications 34, 1 (2014), 77–80.

[5]

Lars Bretzner, Ivan Laptev, and Tony Lindeberg. 2002. Hand Gesture Recognition using Multi-Scale Colour Features, Hierarchical Models and Particle Filtering. In 5th IEEE International Conference on Automatic Face and Gesture Recognition (FGR 2002), with CD-ROM, 20-21 May 2002, Washington, D.C., USA. 423–428. https://doi.org/10.1109/AFGR.2002.1004190

[6]

Volkert Buchmann, Stephen Violich, Mark Billinghurst, and Andy Cockburn. 2004. FingARtips: gesture based direct manipulation in Augmented Reality. In Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia. ACM, 212–221.

Digital Library

[7]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2016. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. CoRR abs/1611.08050(2016). arxiv:1611.08050http://arxiv.org/abs/1611.08050

[8]

Han Joo Chae, Jeongin Hwang, and Jinwook Seo. 2018. Wall-based Space Manipulation Technique for Efficient Placement of Distant Objects in Augmented Reality. In The 31st Annual ACM Symposium on User Interface Software and Technology, UIST 2018, Berlin, Germany, October 14-17, 2018. 45–52. https://doi.org/10.1145/3242587.3242631

Digital Library

[9]

Wendy H Chun and Tobias Höllerer. 2013. Real-time hand interaction for augmented reality on mobile phones. In Proceedings of the 2013 international conference on Intelligent user interfaces. ACM, 307–314.

Digital Library

[10]

Buti Al Delail, Luis Weruaga, and M Jamal Zemerly. 2012. CAViAR: Context aware visual indoor augmented reality for a university campus. In Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology-Volume 03. IEEE Computer Society, 286–290.

Digital Library

[11]

Tiare Feuchtner and Jörg Müller. 2017. Extending the body for interaction with reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5145–5157.

Digital Library

[12]

Euan Freeman, Stephen Brewster, and Vuokko Lantz. 2016. Do that, there: an interaction technique for addressing in-air gesture systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2319–2331.

Digital Library

[13]

Ruchi Manish Gurav and Premanand K Kadbe. 2015. Real time finger tracking and contour detection for gesture recognition using OpenCV. In 2015 International Conference on Industrial Instrumentation and Control (ICIC). IEEE, 974–977.

[14]

Taejin Ha, Steven Feiner, and Woontack Woo. 2014. WeARHand: Head-worn, RGB-D camera-based, bare-hand user interface with visually enhanced depth perception. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on. IEEE, 219–228.

[15]

S. G. Hart and L. E. Staveland. 1988. Human Mental Workload. North Holland Press, Amsterdam, Chapter Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research.

[16]

Jeremy Hartmann and Daniel Vogel. 2018. An Evaluation of Mobile Phone Pointing in Spatial Augmented Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, LBW122.

[17]

Otmar Hilliges, David Kim, Shahram Izadi, Malte Weiss, and Andrew Wilson. 2012. HoloDesk: direct 3d interactions with a situated see-through display. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2421–2430.

Digital Library

[18]

Juan David Hincapié-Ramos, Kasim Ozacar, Pourang P Irani, and Yoshifumi Kitamura. 2015. GyroWand: IMU-based raycasting for augmented reality head-mounted displays. In Proceedings of the 3rd ACM Symposium on Spatial User Interaction. ACM, 89–98.

Digital Library

[19]

Feng Huang, Yu Zhou, Yao Yu, Ziqiang Wang, and Sidan Du. 2011. Piano ar: A markerless augmented reality based piano teaching system. In Intelligent Human-Machine Systems and Cybernetics (IHMSC), 2011 International Conference on, Vol. 2. IEEE, 47–52.

Digital Library

[20]

Wolfgang Hürst and Casper Van Wezel. 2013. Gesture-based interaction via finger tracking for mobile augmented reality. Multimedia Tools and Applications 62, 1 (2013), 233–258.

[21]

Christophe Hurter, Audrey Girouard, Nathalie Riche, and Catherine Plaisant. 2011. Active progress bars: facilitating the switch to temporary activities. In CHI’11 Extended Abstracts on Human Factors in Computing Systems. ACM, 1963–1968.

[22]

Sylvia Irawati, Scott Green, Mark Billinghurst, Andreas Duenser, and Heedong Ko. 2006. ” Move the couch where?”: developing an augmented reality multimodal interface. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on. IEEE, 183–186.

Digital Library

[23]

Michael J Jones and James M Rehg. 2002. Statistical color models with application to skin detection. International Journal of Computer Vision 46, 1 (2002), 81–96.

Digital Library

[24]

Neel Joshi, Abhishek Kar, and Michael Cohen. 2012. Looking at you: fused gyro and face tracking for viewing large imagery on mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2211–2220.

Digital Library

[25]

David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 167–176.

Digital Library

[26]

Minseok Kim and Jae Yeol Lee. 2016. Touch and hand gesture-based interactions for directly manipulating 3D virtual objects in mobile augmented reality. Multimedia Tools and Applications 75, 23 (2016), 16529–16550.

Digital Library

[27]

Ernst Kruijff, J Edward Swan, and Steven Feiner. 2010. Perceptual issues in augmented reality revisited. In 2010 IEEE International Symposium on Mixed and Augmented Reality. IEEE, 3–12.

[28]

Taehee Lee and Tobias Hollerer. 2007. Handy AR: Markerless inspection of augmented reality objects using fingertip tracking. (2007).

[29]

Yue Liu, Ju Yang, and Mingjun Liu. 2008. Recognition of QR Code with mobile phones. In 2008 Chinese control and decision conference. IEEE, 203–206.

[30]

Zhihan Lv, Alaa Halawani, Shengzhong Feng, Shafiq Ur Réhman, and Haibo Li. 2015. Touch-less interactive augmented reality game on vision-based wearable device. Personal and Ubiquitous Computing 19, 3-4 (2015), 551–567.

Digital Library

[31]

Sotiris Makris, Panagiotis Karagiannis, Spyridon Koukas, and Aleksandros-Stereos Matthaiakis. 2016. Augmented reality system for operator support in human–robot collaborative assembly. CIRP Annals-Manufacturing Technology 65, 1 (2016), 61–64.

[32]

Shreyash Mohatta, Ramakrishna Perla, Gaurav Gupta, Ehtesham Hassan, and Ramya Hebbalaguppe. 2017. Robust hand gestural interaction for smartphone based AR/VR applications. In Applications of Computer Vision (WACV), 2017 IEEE Winter Conference on. IEEE, 330–335.

[33]

Ryota Nomura, Yuko Unuma, Takashi Komuro, Shoji Yamamoto, and Norimichi Tsumura. 2017. Mobile augmented reality for providing perception of materials. In Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia. ACM, 501–506.

Digital Library

[34]

Kenton O’Hara, Gerardo Gonzalez, Abigail Sellen, Graeme Penney, Andreas Varnavas, Helena Mentis, Antonio Criminisi, Robert Corish, Mark Rouncefield, Neville Dastur, 2014. Touchless interaction in surgery. Commun. ACM 57, 1 (2014), 70–77.

Digital Library

[35]

Jing Qian, Laurent Denoue, Jacob Biehl, and David A Shamma. 2018. AI for Toggling the Linearity of Interactions in AR. In 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). IEEE, 185–186.

[36]

Jing Qian, Jiaju Ma, Xiangyu L, Benjamin Attal, Haoming Lai, James Tompkin, John F. Hughes, and Jeff Huang. 2019. Perceptual issues in augmented reality revisited. In Proceedings of the 32th Annual Symposium on User Interface Software and Technology. ACM.

[37]

Jing Qian, Jiaju Ma, Xiangyu Li, Benjamin Attal, Haoming Lai, James Tompkin, John F Hughes, and Jeff Huang. 2019. Portal-ble: Intuitive Free-hand Manipulation in Unbounded Smartphone-based Augmented Reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 133–145.

Digital Library

[38]

Rafael Radkowski and Christian Stritzke. 2012. Interactive hand gesture-based assembly for augmented reality applications. In Proceedings of the 2012 International Conference on Advances in Computer-Human Interactions. Citeseer, 303–308.

[39]

SE Reed, O Kreylos, S Hsi, LH Kellogg, G Schladow, MB Yikilmaz, H Segale, J Silverman, S Yalowitz, and E Sato. 2014. Shaping watersheds exhibit: An interactive, augmented reality sandbox for advancing earth science education. In AGU Fall Meeting Abstracts.

[40]

Stefan Reifinger, Frank Wallhoff, Markus Ablassmeier, Tony Poitschke, and Gerhard Rigoll. 2007. Static and dynamic hand-gesture recognition for augmented reality applications. In International Conference on Human-Computer Interaction. Springer, 728–737.

[41]

Gerhard Reitmayr and Tom Drummond. 2006. Going out: robust model-based tracking for outdoor augmented reality. In Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality. IEEE Computer Society, 109–118.

Digital Library

[42]

Hyocheol Ro, Seungho Chae, Inhwan Kim, Junghyun Byun, Yoonsik Yang, Yoonjung Park, and Tackdon Han. 2017. A dynamic depth-variable ray-casting interface for object manipulation in ar environments. In Systems, Man, and Cybernetics (SMC), 2017 IEEE International Conference on. IEEE, 2873–2878.

[43]

Franziska Roesner, Tadayoshi Kohno, and David Molnar. 2014. Security and privacy for augmented reality systems. Commun. ACM 57, 4 (2014), 88–96.

Digital Library

[44]

Peter Schmutz, Silvia Heinz, Yolanda Métrailler, and Klaus Opwis. 2009. Cognitive load in ecommerce applications: measurement and effects on user satisfaction. Advances in Human-Computer Interaction 2009 (2009), 3.

Digital Library

[45]

Rajinder S Sodhi, Brett R Jones, David Forsyth, Brian P Bailey, and Giuliano Maciocci. 2013. BeThere: 3D mobile collaboration with spatial input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 179–188.

Digital Library

[46]

Martin Spindler, Martin Schuessler, Marcel Martsch, and Raimund Dachselt. 2014. Pinch-drag-flick vs. spatial input: rethinking zoom & pan on mobile displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1113–1122.

Digital Library

[47]

Kashmiri Stec and Lars Bo Larsen. 2018. Gestures for Controlling a Moveable TV. In Proceedings of the 2018 ACM International Conference on Interactive Experiences for TV and Online Video. 5–14.

Digital Library

[48]

Arthur Tang, Charles Owen, Frank Biocca, and Weimin Mou. 2003. Comparative effectiveness of augmented reality in object assembly. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 73–80.

Digital Library

[49]

Takahiro Uchiya, Masaki Yoshida, Daisuke Yamamoto, Ryota Nishimura, and Ichi Takumi. 2015. Design and Implementation of Open Campus Event System with Voice Interaction Agent.J. Mobile Multimedia 11, 3&4 (2015), 237–250.

Digital Library

[50]

Matt Whitlock, Ethan Harnner, Jed R Brubaker, Shaun Kane, and Danielle Albers Szafir. 2018. Interacting with Distant Objects in Augmented Reality. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 41–48.

[51]

Jinghong Xiong and Satoshi Muraki. 2016. Effects of age, thumb length and screen size on thumb movement coverage on smartphone touchscreens. International Journal of Industrial Ergonomics 53 (2016), 140–148.

Cited By

Wu GQian JCastelo Quispe SChen SRulff JSilva C(2024)ARTiST: Automated Text Simplification for Task Guidance in Augmented RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642772(1-24)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642772
Batistão LSementille ACorrêa CSanches S(2024)Pinch gesture interaction in the peripersonal space using VST smartphone-based HMDsMultimedia Tools and Applications10.1007/s11042-024-18736-583:34(80873-80887)Online publication date: 9-Mar-2024
https://doi.org/10.1007/s11042-024-18736-5
Ma JQian JZhou THuang J(2023)FocalPointProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808567:1(1-26)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3580856
Show More Cited By

Recommendations

Personalizing 3D Free-Hand Input for Intuitive Smartphone Augmented Reality Interactions
UIST '20 Adjunct: Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

My research goal is to build practical and intuitive 3D free-hand interactions for augmented reality (AR) on smartphones and to explore relevant behavioral-data-driven interaction techniques. In service of that goal, I have developed two preliminary AR ...
Enable Natural User Interactions in Handheld Mobile Augmented Reality through Image Computing
SUI '24: Proceedings of the 2024 ACM Symposium on Spatial User Interaction

Augmented Reality (AR) has transformed user interaction with the digital world by blending virtual elements with the physical environment. While various AR interaction methods have been studied, most research has concentrated on evaluating interactions ...
Occlusion based interaction methods for tangible augmented reality environments
VRCAI '04: Proceedings of the 2004 ACM SIGGRAPH international conference on Virtual Reality continuum and its applications in industry

Traditional Tangible Augmented Reality (Tangible AR) interfaces combine a mixture of tangible user interface and augmented reality technology, complementing each other for novel interaction methods and real world anchored visualization. However, well ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IMX '20: Proceedings of the 2020 ACM International Conference on Interactive Media Experiences

June 2020

211 pages

ISBN:9781450379762

DOI:10.1145/3391614

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

IMX '20

Sponsor:

IMX '20: ACM International Conference on Interactive Media Experiences

June 17 - 19, 2020

Cornella, Barcelona, Spain

Acceptance Rates

Overall Acceptance Rate 69 of 245 submissions, 28%

Upcoming Conference

IMX '25

Sponsor:
sigchi

ACM International Conference on Interactive Media Experiences

June 3 - 6, 2025

Niter?i , Brazil

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
457
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu GQian JCastelo Quispe SChen SRulff JSilva C(2024)ARTiST: Automated Text Simplification for Task Guidance in Augmented RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642772(1-24)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642772
Batistão LSementille ACorrêa CSanches S(2024)Pinch gesture interaction in the peripersonal space using VST smartphone-based HMDsMultimedia Tools and Applications10.1007/s11042-024-18736-583:34(80873-80887)Online publication date: 9-Mar-2024
https://doi.org/10.1007/s11042-024-18736-5
Ma JQian JZhou THuang J(2023)FocalPointProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808567:1(1-26)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3580856
Spittle BFrutos-Pascual MCreed CWilliams I(2023)A Review of Interaction Techniques for Immersive EnvironmentsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.317480529:9(3900-3921)Online publication date: 1-Sep-2023
https://doi.org/10.1109/TVCG.2022.3174805
Halim NIsmail A(2022)Target selection in handheld Augmented Reality for distant and occluded object2022 IEEE 7th International Conference on Recent Advances and Innovations in Engineering (ICRAIE)10.1109/ICRAIE56454.2022.10054325(266-271)Online publication date: 1-Dec-2022
https://doi.org/10.1109/ICRAIE56454.2022.10054325
Raeburn GTokarchuk L(2021)Varying user agency and interaction opportunities in a home mobile augmented virtuality story2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR52148.2021.00051(347-356)Online publication date: Oct-2021
https://doi.org/10.1109/ISMAR52148.2021.00051
Abdul Halim NIsmail A(2021)Raycasting method using hand gesture for target selection on the occluded object in handheld Augmented Reality2021 6th IEEE International Conference on Recent Advances and Innovations in Engineering (ICRAIE)10.1109/ICRAIE52900.2021.9704035(1-6)Online publication date: 1-Dec-2021
https://doi.org/10.1109/ICRAIE52900.2021.9704035

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten