Jing Qian
David A. Shamma
Jacob Biehl
ABSTRACT
Augmented reality (AR) on smartphone devices allows people to interact with virtually placed objects anchored in the real world through the device’s viewport. Typically, smartphone AR interactions work with the device’s 2D touchscreen disconnected from the modality and depth of the virtual objects. In this paper, we studied 15 participants’ preferences, performances, and cognitive loads on a set of common tasks performed on smartphones at two interaction depths (close-range and distant) with two touchless modalities (hand tracking and screen dwell). We find that distant AR interactions, strongly preferred by the participants, were significantly faster and took less cognitive effort. We observed that within both interaction depths, modalities performed equally. When designing touchless modalities on smartphones, we suggest using distant interactions when overall performance is the top priority, otherwise, using hand tracking or screen dwell as back-ups for each other can be equally effective.
- Roland Aigner, Daniel Wigdor, Hrvoje Benko, Michael Haller, David Lindbauer, Alexandra Ion, Shengdong Zhao, and JTKV Koh. 2012. Understanding mid-air hand gestures: A study of human preferences in usage of gesture types for hci. Microsoft Research TechReport MSR-TR-2012-111 2 (2012).Google Scholar
- Huidong Bai, Gun A Lee, Mukundan Ramakrishnan, and Mark Billinghurst. 2014. 3D gesture interaction for handheld augmented reality. In SIGGRAPH Asia 2014 Mobile Graphics and Interactive Applications. ACM, ACM, 7.Google Scholar
- Evan Barba and Ramon Zamora Marroquin. 2017. A Primer on Spatial Scale and Its Application to Mixed Reality. In Mixed and Augmented Reality (ISMAR), 2017 IEEE International Symposium on. IEEE, IEEE, 100–110.Google ScholarCross Ref
- Mark Billinghurst, Tham Piumsomboon, and Huidong Bai. 2014. Hands in space: Gesture interaction with augmented-reality interfaces. IEEE computer graphics and applications 34, 1 (2014), 77–80.Google Scholar
- Lars Bretzner, Ivan Laptev, and Tony Lindeberg. 2002. Hand Gesture Recognition using Multi-Scale Colour Features, Hierarchical Models and Particle Filtering. In 5th IEEE International Conference on Automatic Face and Gesture Recognition (FGR 2002), with CD-ROM, 20-21 May 2002, Washington, D.C., USA. 423–428. https://doi.org/10.1109/AFGR.2002.1004190Google Scholar
- Volkert Buchmann, Stephen Violich, Mark Billinghurst, and Andy Cockburn. 2004. FingARtips: gesture based direct manipulation in Augmented Reality. In Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia. ACM, 212–221.Google ScholarDigital Library
- Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2016. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. CoRR abs/1611.08050(2016). arxiv:1611.08050http://arxiv.org/abs/1611.08050Google Scholar
- Han Joo Chae, Jeongin Hwang, and Jinwook Seo. 2018. Wall-based Space Manipulation Technique for Efficient Placement of Distant Objects in Augmented Reality. In The 31st Annual ACM Symposium on User Interface Software and Technology, UIST 2018, Berlin, Germany, October 14-17, 2018. 45–52. https://doi.org/10.1145/3242587.3242631Google ScholarDigital Library
- Wendy H Chun and Tobias Höllerer. 2013. Real-time hand interaction for augmented reality on mobile phones. In Proceedings of the 2013 international conference on Intelligent user interfaces. ACM, 307–314.Google ScholarDigital Library
- Buti Al Delail, Luis Weruaga, and M Jamal Zemerly. 2012. CAViAR: Context aware visual indoor augmented reality for a university campus. In Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology-Volume 03. IEEE Computer Society, 286–290.Google ScholarDigital Library
- Tiare Feuchtner and Jörg Müller. 2017. Extending the body for interaction with reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5145–5157.Google ScholarDigital Library
- Euan Freeman, Stephen Brewster, and Vuokko Lantz. 2016. Do that, there: an interaction technique for addressing in-air gesture systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2319–2331.Google ScholarDigital Library
- Ruchi Manish Gurav and Premanand K Kadbe. 2015. Real time finger tracking and contour detection for gesture recognition using OpenCV. In 2015 International Conference on Industrial Instrumentation and Control (ICIC). IEEE, 974–977.Google ScholarCross Ref
- Taejin Ha, Steven Feiner, and Woontack Woo. 2014. WeARHand: Head-worn, RGB-D camera-based, bare-hand user interface with visually enhanced depth perception. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on. IEEE, 219–228.Google ScholarCross Ref
- S. G. Hart and L. E. Staveland. 1988. Human Mental Workload. North Holland Press, Amsterdam, Chapter Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research.Google Scholar
- Jeremy Hartmann and Daniel Vogel. 2018. An Evaluation of Mobile Phone Pointing in Spatial Augmented Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, LBW122.Google Scholar
- Otmar Hilliges, David Kim, Shahram Izadi, Malte Weiss, and Andrew Wilson. 2012. HoloDesk: direct 3d interactions with a situated see-through display. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2421–2430.Google ScholarDigital Library
- Juan David Hincapié-Ramos, Kasim Ozacar, Pourang P Irani, and Yoshifumi Kitamura. 2015. GyroWand: IMU-based raycasting for augmented reality head-mounted displays. In Proceedings of the 3rd ACM Symposium on Spatial User Interaction. ACM, 89–98.Google ScholarDigital Library
- Feng Huang, Yu Zhou, Yao Yu, Ziqiang Wang, and Sidan Du. 2011. Piano ar: A markerless augmented reality based piano teaching system. In Intelligent Human-Machine Systems and Cybernetics (IHMSC), 2011 International Conference on, Vol. 2. IEEE, 47–52.Google ScholarDigital Library
- Wolfgang Hürst and Casper Van Wezel. 2013. Gesture-based interaction via finger tracking for mobile augmented reality. Multimedia Tools and Applications 62, 1 (2013), 233–258.Google ScholarCross Ref
- Christophe Hurter, Audrey Girouard, Nathalie Riche, and Catherine Plaisant. 2011. Active progress bars: facilitating the switch to temporary activities. In CHI’11 Extended Abstracts on Human Factors in Computing Systems. ACM, 1963–1968.Google Scholar
- Sylvia Irawati, Scott Green, Mark Billinghurst, Andreas Duenser, and Heedong Ko. 2006. ” Move the couch where?”: developing an augmented reality multimodal interface. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on. IEEE, 183–186.Google ScholarDigital Library
- Michael J Jones and James M Rehg. 2002. Statistical color models with application to skin detection. International Journal of Computer Vision 46, 1 (2002), 81–96.Google ScholarDigital Library
- Neel Joshi, Abhishek Kar, and Michael Cohen. 2012. Looking at you: fused gyro and face tracking for viewing large imagery on mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2211–2220.Google ScholarDigital Library
- David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 167–176.Google ScholarDigital Library
- Minseok Kim and Jae Yeol Lee. 2016. Touch and hand gesture-based interactions for directly manipulating 3D virtual objects in mobile augmented reality. Multimedia Tools and Applications 75, 23 (2016), 16529–16550.Google ScholarDigital Library
- Ernst Kruijff, J Edward Swan, and Steven Feiner. 2010. Perceptual issues in augmented reality revisited. In 2010 IEEE International Symposium on Mixed and Augmented Reality. IEEE, 3–12.Google ScholarCross Ref
- Taehee Lee and Tobias Hollerer. 2007. Handy AR: Markerless inspection of augmented reality objects using fingertip tracking. (2007).Google Scholar
- Yue Liu, Ju Yang, and Mingjun Liu. 2008. Recognition of QR Code with mobile phones. In 2008 Chinese control and decision conference. IEEE, 203–206.Google Scholar
- Zhihan Lv, Alaa Halawani, Shengzhong Feng, Shafiq Ur Réhman, and Haibo Li. 2015. Touch-less interactive augmented reality game on vision-based wearable device. Personal and Ubiquitous Computing 19, 3-4 (2015), 551–567.Google ScholarDigital Library
- Sotiris Makris, Panagiotis Karagiannis, Spyridon Koukas, and Aleksandros-Stereos Matthaiakis. 2016. Augmented reality system for operator support in human–robot collaborative assembly. CIRP Annals-Manufacturing Technology 65, 1 (2016), 61–64.Google ScholarCross Ref
- Shreyash Mohatta, Ramakrishna Perla, Gaurav Gupta, Ehtesham Hassan, and Ramya Hebbalaguppe. 2017. Robust hand gestural interaction for smartphone based AR/VR applications. In Applications of Computer Vision (WACV), 2017 IEEE Winter Conference on. IEEE, 330–335.Google ScholarCross Ref
- Ryota Nomura, Yuko Unuma, Takashi Komuro, Shoji Yamamoto, and Norimichi Tsumura. 2017. Mobile augmented reality for providing perception of materials. In Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia. ACM, 501–506.Google ScholarDigital Library
- Kenton O’Hara, Gerardo Gonzalez, Abigail Sellen, Graeme Penney, Andreas Varnavas, Helena Mentis, Antonio Criminisi, Robert Corish, Mark Rouncefield, Neville Dastur, 2014. Touchless interaction in surgery. Commun. ACM 57, 1 (2014), 70–77.Google ScholarDigital Library
- Jing Qian, Laurent Denoue, Jacob Biehl, and David A Shamma. 2018. AI for Toggling the Linearity of Interactions in AR. In 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). IEEE, 185–186.Google Scholar
- Jing Qian, Jiaju Ma, Xiangyu L, Benjamin Attal, Haoming Lai, James Tompkin, John F. Hughes, and Jeff Huang. 2019. Perceptual issues in augmented reality revisited. In Proceedings of the 32th Annual Symposium on User Interface Software and Technology. ACM.Google Scholar
- Jing Qian, Jiaju Ma, Xiangyu Li, Benjamin Attal, Haoming Lai, James Tompkin, John F Hughes, and Jeff Huang. 2019. Portal-ble: Intuitive Free-hand Manipulation in Unbounded Smartphone-based Augmented Reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 133–145.Google ScholarDigital Library
- Rafael Radkowski and Christian Stritzke. 2012. Interactive hand gesture-based assembly for augmented reality applications. In Proceedings of the 2012 International Conference on Advances in Computer-Human Interactions. Citeseer, 303–308.Google Scholar
- SE Reed, O Kreylos, S Hsi, LH Kellogg, G Schladow, MB Yikilmaz, H Segale, J Silverman, S Yalowitz, and E Sato. 2014. Shaping watersheds exhibit: An interactive, augmented reality sandbox for advancing earth science education. In AGU Fall Meeting Abstracts.Google Scholar
- Stefan Reifinger, Frank Wallhoff, Markus Ablassmeier, Tony Poitschke, and Gerhard Rigoll. 2007. Static and dynamic hand-gesture recognition for augmented reality applications. In International Conference on Human-Computer Interaction. Springer, 728–737.Google ScholarCross Ref
- Gerhard Reitmayr and Tom Drummond. 2006. Going out: robust model-based tracking for outdoor augmented reality. In Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality. IEEE Computer Society, 109–118.Google ScholarDigital Library
- Hyocheol Ro, Seungho Chae, Inhwan Kim, Junghyun Byun, Yoonsik Yang, Yoonjung Park, and Tackdon Han. 2017. A dynamic depth-variable ray-casting interface for object manipulation in ar environments. In Systems, Man, and Cybernetics (SMC), 2017 IEEE International Conference on. IEEE, 2873–2878.Google Scholar
- Franziska Roesner, Tadayoshi Kohno, and David Molnar. 2014. Security and privacy for augmented reality systems. Commun. ACM 57, 4 (2014), 88–96.Google ScholarDigital Library
- Peter Schmutz, Silvia Heinz, Yolanda Métrailler, and Klaus Opwis. 2009. Cognitive load in ecommerce applications: measurement and effects on user satisfaction. Advances in Human-Computer Interaction 2009 (2009), 3.Google ScholarDigital Library
- Rajinder S Sodhi, Brett R Jones, David Forsyth, Brian P Bailey, and Giuliano Maciocci. 2013. BeThere: 3D mobile collaboration with spatial input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 179–188.Google ScholarDigital Library
- Martin Spindler, Martin Schuessler, Marcel Martsch, and Raimund Dachselt. 2014. Pinch-drag-flick vs. spatial input: rethinking zoom & pan on mobile displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1113–1122.Google ScholarDigital Library
- Kashmiri Stec and Lars Bo Larsen. 2018. Gestures for Controlling a Moveable TV. In Proceedings of the 2018 ACM International Conference on Interactive Experiences for TV and Online Video. 5–14.Google ScholarDigital Library
- Arthur Tang, Charles Owen, Frank Biocca, and Weimin Mou. 2003. Comparative effectiveness of augmented reality in object assembly. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 73–80.Google ScholarDigital Library
- Takahiro Uchiya, Masaki Yoshida, Daisuke Yamamoto, Ryota Nishimura, and Ichi Takumi. 2015. Design and Implementation of Open Campus Event System with Voice Interaction Agent.J. Mobile Multimedia 11, 3&4 (2015), 237–250.Google ScholarDigital Library
- Matt Whitlock, Ethan Harnner, Jed R Brubaker, Shaun Kane, and Danielle Albers Szafir. 2018. Interacting with Distant Objects in Augmented Reality. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 41–48.Google Scholar
- Jinghong Xiong and Satoshi Muraki. 2016. Effects of age, thumb length and screen size on thumb movement coverage on smartphone touchscreens. International Journal of Industrial Ergonomics 53 (2016), 140–148.Google ScholarCross Ref
Recommendations
Personalizing 3D Free-Hand Input for Intuitive Smartphone Augmented Reality Interactions
UIST '20 Adjunct: Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and TechnologyMy research goal is to build practical and intuitive 3D free-hand interactions for augmented reality (AR) on smartphones and to explore relevant behavioral-data-driven interaction techniques. In service of that goal, I have developed two preliminary AR ...
Occlusion based interaction methods for tangible augmented reality environments
VRCAI '04: Proceedings of the 2004 ACM SIGGRAPH international conference on Virtual Reality continuum and its applications in industryTraditional Tangible Augmented Reality (Tangible AR) interfaces combine a mixture of tangible user interface and augmented reality technology, complementing each other for novel interaction methods and real world anchored visualization. However, well ...
Evaluating Menu Techniques for Handheld AR with a Smartphone & Mid-Air Pen
MobileHCI '20: 22nd International Conference on Human-Computer Interaction with Mobile Devices and ServicesAdding a mid-air pen to Handheld Augmented Reality creates a new kind of bimanual interaction for which many fundamental interaction design questions have not been answered yet. In particular, menus are an essential component in most visual interfaces, ...
Comments