skip to main content
10.1145/3339825.3391870acmconferencesArticle/Chapter ViewAbstractPublication PagesmmsysConference Proceedingsconference-collections
research-article

UbiPoint: towards non-intrusive mid-air interaction for hardware constrained smart glasses

Published: 27 May 2020 Publication History

Abstract

Throughout the past decade, numerous interaction techniques have been designed for mobile and wearable devices. Among these devices, smartglasses mostly rely on hardware interfaces such as touchpad and buttons, which are often cumbersome and counterintuitive to use. Furthermore, smartglasses feature cheap and low-power hardware preventing the use of advanced pointing techniques. To overcome these issues, we introduce UbiPoint, a freehand mid-air interaction technique. UbiPoint uses the monocular camera embedded in smartglasses to detect the user's hand without relying on gloves, markers, or sensors, enabling intuitive and non-intrusive interaction. We introduce a computationally fast and light-weight algorithm for fingertip detection, which is especially suited for the limited hardware specifications and the short battery life of smartglasses. UbiPoint processes pictures at a rate of 20 frames per second with high detection accuracy - no more than 6 pixels deviation. Our evaluation shows that UbiPoint, as a mid-air non-intrusive interface, delivers a better experience for users and smart glasses interactions, with users completing typical tasks 1.82 times faster than when using the original hardware.

References

[1]
M. Abramowitz and I.A. Stegun. 1972. Handbook of mathematical functions: with formulas, graphs, and mathematical tables. Courier Dover Publications (55).
[2]
Gilles Bailly, Jörg Müller, Michael Rohs, Daniel Wigdor, and Sven Kratz. 2012. ShoeSense:A New Perspective on Gestural Interaction and Wearable Applications. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CHI '12). ACM, 1239--1248.
[3]
W. Buxton and B. Myers. 1986. A Study in Two-handed Input. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (Boston, Massachusetts, USA) (CHI '86).
[4]
Hang Zhi Cheng. 2017. http://www.thetechrevolutionist.com/2016/01/review-of-epson-moverio-bt-200-vrar.html. [Online; accessed 10-February-2017].
[5]
Epson. 2017. Epson Moverio BT-200. http://www.epson.com/cgi-bin/Store/jsp/Landing/moverio-augmented-reality-smart-glasses.do?UseCookie=yes
[6]
Goel et al. 2015. Tongue-in-Cheek: Using Wireless Signals to Enable Non-Intrusive and Flexible Facial Gestures Detection. In Proc. of the 33rd Annual ACM Conf. on Human Factors in Computing Systems (CHI '15).
[7]
Georgiana Simion et al. 2016. Fingertip-based real time tracking and gesture recognition for natural user interfaces. 13 (01 2016), 189--204.
[8]
Istance et al. 2008. Snap Clutch, a Moded Approach to Solving the Midas Touch Problem. 38 (2008).
[9]
Jacek et al. 2016. Performance Analysis of Interaction between Smart Glasses and Smart Objects Using Image-Based Object Identification. Int. Journal of Distributed Sensor Networks 12, 3 (2016), 10--27.
[10]
P. Hamette et al. 2002. FingerMouse: A Wearable Hand Tracking System. (01 2002).
[11]
R. Gang et al. 2013. 3D selection with freehand gesture. Computers & Graphics 37, 3 (2013), 101--120.
[12]
Schuchert et al. 2012. Sensing Visual Attention Using an Interactive Bidirectional HMD. In Proc. of the 4th Workshop on Eye Gaze in Intelligent Human Machine Interaction (Gaze-In '12). ACM, Article 16, 3 pages.
[13]
Wang et al. 2015. PalmType: Using Palms As Keyboards for Smart Glasses. In Proc. of the 17th Int. Conf. on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, 153--160.
[14]
Yu et al. 2016. One-Dimensional Handwriting: Inputting Letters and Words on Smart Glasses. In Proc. of the 2016 CHI Conf. on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). ACM, 71--82.
[15]
Yichao Huang et al. 2016. A Pointing Gesture Based Egocentric Interaction System: Dataset, Approach and Application. (2016).
[16]
Y.-T. Hsieh et al. 2016. Designing a Willing-to-Use-in-Public Hand Gestural Interaction Technique for Smart Glasses. In Proc. of the 2016 CHI Conf. on Human Factors in Computing Systems (CHI '16).
[17]
Zheng et al. 2015. Eye-Wearable Technology for Machine Maintenance: Effects of Display Position and Hands-free Operation. In Proc. of the 33rd Annual ACM Conf. on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2125--2134.
[18]
Alphabet Inc. (Google). 2017. Google Glass. https://developers.google.com/glass/
[19]
Faizan Haque, Mathieu Nancel, and Daniel Vogel. 2015. Myopoint: Pointing and Clicking Using Forearm Mounted Electromyography and Inertial Motion Sensors. In Proc. of the 33rd Annual ACM Conf. on Human Factors in Computing Systems (CHI '15). ACM, 3653--3656.
[20]
Z. Huang, W. Li, and P. Hui. 2015. Ubii: Towards Seamless Interaction Between Digital and Physical Worlds. In Proc. of the 23rd ACM Int. Conf. on Multimedia (MM '15).
[21]
Robert J.K. Jacob, Audrey Girouard, Leanne M. Hirshfield, Michael S. Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-based Interaction: A Framework for post-WIMP Interfaces. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 201--210.
[22]
Eleanor Jones, Jason Alexander, Andreas Andreou, Pourang Irani, and Sriram Subramanian. 2010. GesText: Accelerometer-based Gestural Text-entry Systems. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CHI '10). ACM, 2173--2182.
[23]
S. Lawrence and W. Brett. 2013. Comparison of Gestural, Touch, and Mouse Interaction with Fitts' Law. In Proc. of the 25th Australian Computer-Human Interaction Conf.: Augmentation, Application, Innovation, Collaboration (OzCHI '13). ACM, 119--122.
[24]
Lik Hang Lee, Tristan Braud, Farshid Hassani Bijarbooneh, and Pan Hui. 2019. TiPoint: Detecting Fingertip for Mid-Air Interaction on Computational Resource Constrained Smartglasses. In Proceedings of the 23rd International Symposium on Wearable Computers (London, United Kingdom) (ISWC '19). Association for Computing Machinery, New York, NY, USA, 118--122.
[25]
Lik Hang Lee, Tristan Braud, Kit Yung Lam, Yui Pan Yau, and Pan Hui. 2020. From seen to unseen: Designing keyboard-less interfaces for text entry on the constrained screen real estate of Augmented Reality headsets. Pervasive and Mobile Computing 64 (2020), 101148.
[26]
Lik Hang Lee and Pan Hui. 2018. Interaction Methods for Smart Glasses: A Survey. IEEE Access 6 (2018), 28712--32.
[27]
Lik Hang Lee, Kit Yung Lam, Tong Li, Tristan Braud, Xiang Su, and Pan Hui. 2019. Quadmetric Optimized Thumb-to-Finger Interaction for Force Assisted One-Handed Text Entry on Mobile Headsets. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 3, Article 94 (Sept. 2019), 27 pages.
[28]
L. H. Lee, K. Yung Lam, Y. P. Yau, T. Braud, and P. Hui. 2019. HIBEY: Hide the Keyboard in Augmented Reality. In 2019 IEEE International Conference on Pervasive Computing and Communications (PerCom). 1--10.
[29]
L. H. Lee, Y. Zhu, Y. P. Yau, T. Braud, X. Su, and P Hui. 2020. One-thumb Text Acquisition on Force-assisted Miniature Interfaces for Mobile Headsets. In 2020 IEEE International Conference on Pervasive Computing and Communications (PerCom). 1--10.
[30]
T. Lee and T. Hollerer. 2007. Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking. In 2007 11th IEEE Int. Sym. on Wearable Computers. 83--90.
[31]
I Scott MacKenzie. 1992. Fitts' law as a research and design tool in humancomputer interaction. Human-computer interaction 7, 1 (1992), 91--139.
[32]
Microsoft. 2017. Microsoft Hololens. https://www.microsoft.com/microsoft-hololens/en-us
[33]
K. Sobottka and I. Pitas. 1998. A novel method for automatic face segmentation, facial feature extraction and tracking. Signal Processing: Image Communication 12, 3 (1998), 263 -- 281.
[34]
T. E. Starner. 2002. Attention, memory, and wearable interfaces. IEEE Pervasive Computing 1, 4 (Oct 2002).
[35]
Ying-Chao Tung, Chun-Yen Hsu, Han-Yu Wang, Silvia Chyou, Jhe-Wei Lin, Pei-Jung Wu, Andries Valstar, and Mike Y. Chen. 2015. User-Defined Game Input for Smart Glasses in Public Space. In Proc. of the 33rd Annual ACM Conf. on Human Factors in Computing Systems (CHI '15). ACM, 3327--3336.
[36]
F. Vernier and L. Nigay. 2001. A Framework for the Combination and Characterization of Output Modalities. In Proc. of the 7th Int. Conf. on Design, Specification, and Verification of Interactive Systems (Limerick, Ireland) (DSV-IS'00). Springer-Verlag, 35--50.
[37]
Shanhe Yi, Zhengrui Qin, E Novak, Y Yin, and Q Li. 2016. Glassgesture: Exploring head gesture interface of smart glasses. INFOCOM, 2016 Proc. IEEE (2016).
[38]
Sang Ho Yoon, Ke Huo, Vinh P. Nguyen, and Karthik Ramani. 2015. TIMMi: Finger-worn Textile Input Device with Multimodal Sensing in Mobile Interaction. In Proc. of the Ninth Int. Conf. on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, 269--272.

Cited By

View all
  • (2024)Faraway, so Close: Perceptions of the Metaverse on the Edge of MadnessComputers10.3390/computers1301001913:1(19)Online publication date: 8-Jan-2024
  • (2023)Usability and User Experience Evaluation of Touchable Holographic Solutions: A Systematic Mapping StudyProceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems10.1145/3638067.3638071(1-13)Online publication date: 16-Oct-2023
  • (2023)Optimization in Mobile Augmented Reality Systems for the Metaverse Over Wireless CommunicationsGLOBECOM 2023 - 2023 IEEE Global Communications Conference10.1109/GLOBECOM54140.2023.10437658(5439-5444)Online publication date: 4-Dec-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MMSys '20: Proceedings of the 11th ACM Multimedia Systems Conference
May 2020
403 pages
ISBN:9781450368452
DOI:10.1145/3339825
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2020

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article

Funding Sources

  • Academy of Finland
  • Research Grants Council of Hong Kong

Conference

MMSys '20
Sponsor:
MMSys '20: 11th ACM Multimedia Systems Conference
June 8 - 11, 2020
Istanbul, Turkey

Acceptance Rates

MMSys '20 Paper Acceptance Rate 18 of 55 submissions, 33%;
Overall Acceptance Rate 176 of 530 submissions, 33%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)24
  • Downloads (Last 6 weeks)1
Reflects downloads up to 06 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Faraway, so Close: Perceptions of the Metaverse on the Edge of MadnessComputers10.3390/computers1301001913:1(19)Online publication date: 8-Jan-2024
  • (2023)Usability and User Experience Evaluation of Touchable Holographic Solutions: A Systematic Mapping StudyProceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems10.1145/3638067.3638071(1-13)Online publication date: 16-Oct-2023
  • (2023)Optimization in Mobile Augmented Reality Systems for the Metaverse Over Wireless CommunicationsGLOBECOM 2023 - 2023 IEEE Global Communications Conference10.1109/GLOBECOM54140.2023.10437658(5439-5444)Online publication date: 4-Dec-2023
  • (2023)The Metaverse: Applications, Concerns, Technical Challenges, Future Directions and RecommendationsIEEE Access10.1109/ACCESS.2023.332165011(110850-110861)Online publication date: 2023
  • (2022)EdgeXAR: A 6-DoF Camera Multi-target Interaction Framework for MAR with User-friendly Latency CompensationProceedings of the ACM on Human-Computer Interaction10.1145/35322026:EICS(1-24)Online publication date: 17-Jun-2022
  • (2022)3DeformRProceedings of the 13th ACM Multimedia Systems Conference10.1145/3524273.3528180(52-61)Online publication date: 14-Jun-2022
  • (2022)PassWalk: Spatial Authentication Leveraging Lateral Shift and Gaze on Mobile HeadsetsProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3548252(952-960)Online publication date: 10-Oct-2022
  • (2022)Re-shaping Post-COVID-19 Teaching and Learning: A Blueprint of Virtual-Physical Blended Classrooms in the Metaverse Era2022 IEEE 42nd International Conference on Distributed Computing Systems Workshops (ICDCSW)10.1109/ICDCSW56584.2022.00053(241-247)Online publication date: Jul-2022
  • (2021)Applications of Smart Glasses in Applied Sciences: A Systematic ReviewApplied Sciences10.3390/app1111495611:11(4956)Online publication date: 27-May-2021
  • (2021)Seeing is Believing?Proceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475552(4183-4192)Online publication date: 17-Oct-2021
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media