research-article

Watching Your Phone's Back: Gesture Recognition by Sensing Acoustical Structure-borne Propagation

Authors:

Yuanshuang Jiang,

Daqing ZhangAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 5, Issue 2

Article No.: 82, Pages 1 - 26

https://doi.org/10.1145/3463522

Published: 24 June 2021 Publication History

Abstract

Gesture recognition on the back surface of mobile phone, not limited to the touch screen, is an enabling Human-Computer Interaction (HCI) mechanism which enriches the user interaction experiences. However, there are two main limitations in the existing Back-of-Device (BoD) gesture recognition systems. They can only handle coarse-grained gesture recognition such as tap detection and cannot avoid the air-borne propagation suffering from the interference in the air. In this paper, we propose StruGesture, a fine-grained gesture recognition system using the back of mobile phones with ultrasonic signals. The key technique is to use the structure-borne sounds (i.e., sound propagation via structure of the device) to recognize sliding gestures on the back of mobile phones. StruGesture can fully extract the structure-borne component from the hybrid Channel Impulse Response (CIR) based on Peak Selection Algorithm. We develop a deep adversarial learning architecture to learn the gesture-specific representation for robust and effective recognition. Extensive experiments are designed to evaluate the robustness over nine deployment scenarios. The results show that StruGesture outperforms the competitive state-of-the-art classifiers by achieving an average recognition accuracy of 99.5% over 10 gestures.

References

[1]

Heba Abdelnasser, Moustafa Youssef, and Khaled A Harras. 2015. WiGest: A Ubiquitous WiFi-based Gesture Recognition System. In Proceedings of IEEE INFOCOM.

[2]

Kamran Ali, Alex X. Liu, Wei Wang, and Muhammad Shahzad. 2015. Keystroke Recognition Using WiFi Signals. In Proceedings of ACM MobiCom.

Digital Library

[3]

Md Tanvir Islam Aumi, Sidhant Gupta, Mayank Goel, Eric Larson, and Shwetak Patel. 2013. DopLink: using the doppler effect for multi-device interaction. In Proceedings of ACM UbiComp.

[4]

Ke-Yu Chen, Daniel Ashbrook, Mayank Goel, Sung-Hyuck Lee, and Shwetak Patel. 2014. AirLink: sharing files between multiple devices using in-air gestures. In Proceedings of ACM UbiComp.

Digital Library

[5]

Mingshi Chen, Panlong Yang, Jie Xiong, Maotian Zhang, Youngki Lee, Chaocan Xiang, and Chang Tian. 2019. Your Table Can Be an Input Panel: Acoustic-based Device-Free Interaction Recognition. In Proceedings of ACM UbiComp.

Digital Library

[6]

Emilio Granell and Luis A Leiva. 2017. βap: back-of-device tap input with built-in sensors. In Proceedings of ACM MobileHCI.

[7]

Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Soundwave: Using the Doppler Effect to Sense Gestures. In Proceedings of ACM CHI.

Digital Library

[8]

Chris Harrison, Julia Schwarz, and Scott E Hudson. 2011. TapSense: enhancing finger interaction on touch surfaces. In Proceedings of ACM UIST.

Digital Library

[9]

Franz Hlawatsch and François Auger. 2008. Time-frequency analysis. Wiley Online Library.

[10]

Eva Kollorz, Jochen Penne, Joachim Hornegger, and Alexander Barke. 2008. Gesture recognition with a time-of-flight camera. International Journal of Intelligent Systems Technologies and Applications 5, 3 (2008), 334.

Digital Library

[11]

Huy Viet Le, Sven Mayer, Patrick Bader, and Niels Henze. 2017. A smartphone prototype for touch interaction on the whole device surface. In Proceedings of ACM MobileHCI.

Digital Library

[12]

Kang Ling, Haipeng Dai, Yuntang Liu, Alex X Liu, Wei Wang, and Qing Gu. 2020. Ultragesture: Fine-grained gesture sensing and recognition. IEEE Transactions on Mobile Computing (2020).

[13]

Jian Liu, Yingying Chen, Marco Gruteser, and Yan Wang. 2017. VibSense: Sensing Touches on Ubiquitous Surfaces through Vibration. In Proceedings of IEEE SECON.

Digital Library

[14]

Jian Liu, Chen Wang, Yingying Chen, and Nitesh Saxena. 2017. VibWrite: Towards finger-input authentication on ubiquitous surfaces via physical vibration. In Proceedings of ACM CCS.

Digital Library

[15]

Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proceedings of ACM CHI.

Digital Library

[16]

Chunyi Peng, Guobin Shen, Yongguang Zhang, Yanlin Li, and Kun Tan. 2007. Beepbeep: a High Accuracy Acoustic Ranging System using COTS Mobile Devices. In Proceedings of ACM SenSys.

Digital Library

[17]

Corey R Pittman and Joseph J LaViola Jr. 2017. Multiwave: Complex Hand Gesture Recognition Using the Doppler Effect. In Proceedings of ACM GI.

Digital Library

[18]

A Rodríguez Valiente, A Trinidad, JR García Berrocal, C Górriz, and R Ramírez Camacho. 2014. Extended high-frequency (9-20 kHz) audiometry reference thresholds in 645 healthy subjects. International journal of audiology 53, 8 (2014), 531--545.

[19]

Wenjie Ruan, Quan Z Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: enabling fine-grained hand gesture detection by decoding echo signal. In Proceedings of ACM UbiComp.

Digital Library

[20]

Huihuang Zhang Lili Qiu Sangki Yun, Yi-chao Chen and Wenguang Mao. 2017. Strata: Fined-Grained Device-Free Tracking Using Acoustic Signals. In Proceedings of ACM MobiSys.

[21]

AlsoRF Schaufele and T Shimanouchi. 1967. Longitudinal acoustical vibrations of finite polymethylene chains. The Journal of Chemical Physics 47, 9 (1967), 3605--3610.

[22]

Shaikh Shawon Arefin Shimon, Sarah Morrison-Smith, Noah John, Ghazal Fahimi, and Jaime Ruiz. 2015. Exploring user-defined back-of-device gestures for mobile devices. In Proceedings of ACM MobileHCI.

[23]

Ke Sun, Wei Wang, Alex X Liu, and Haipeng Dai. 2018. Depth aware finger tapping on virtual displays. In Proceedings of ACM MobiSys.

Digital Library

[24]

Ke Sun, Ting Zhao, Wei Wang, and Lei Xie. 2018. Vskin: Sensing touch gestures on surfaces of mobile devices using acoustic signals. In Proceedings of ACM MobiCom.

Digital Library

[25]

Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. In Proceedings of ACM MobiCom.

Digital Library

[26]

Sheng Tan and Jie Yang. 2016. WiFinger: Leveraging Commodity WiFi for Fine-grained Finger Gesture Recognition. In Proceedings of ACM Mobihoc.

Digital Library

[27]

Yu-Chih Tung and Kang G Shin. 2016. Expansion of human-phone interface by sensing structure-borne sound propagation. In Proceedings of ACM MobiSys.

Digital Library

[28]

Chong Wang, Zhong Liu, and Shing-Chow Chan. 2014. Superpixel-based hand gesture recognition with kinect depth camera. IEEE transactions on multimedia 17, 1 (2014), 29--39.

[29]

Lei Wang, Ke Sun, Haipeng Dai, Alex X Liu, and Xiaoyu Wang. 2018. WiTrace: Centimeter-Level Passive Gesture Tracking Using WiFi Signals. In Proceedings of IEEE SECON.

Digital Library

[30]

Wei Wang, Alex X. Liu, and Ke Sun. 2016. Device-free gesture tracking using acoustic signals. In Proceedings of ACM MobiCom.

[31]

Daniel Wigdor, Clifton Forlines, Patrick Baudisch, John Barnwell, and Chia Shen. 2007. Lucid touch: a see-through mobile device. In Proceedings of ACM UIST.

Digital Library

[32]

Pui Chung Wong, Hongbo Fu, and Kening Zhu. 2016. Back-Mirror: back-of-device one-handed interaction on smartphones. In Proceedings of ACM SA.

Digital Library

[33]

Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018. Cbam: Convolutional block attention module. In Proceedings of ECCV. 3--19.

Digital Library

[34]

Xiang Xiao, Teng Han, and Jingtao Wang. 2013. LensGesture: augmenting mobile interactions with back-of-device finger gestures. In Proceedings of ACM ICMI.

Digital Library

[35]

Nan Yu, Wei Wang, Alex X Liu, and Lingtao Kong. 2018. QGesture: Quantifying gesture distance and direction with WiFi signals. In Proceedings of ACM UbiComp.

Digital Library

[36]

Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a mobile device into a mouse in the air. In Proceedings of ACM MobiSys.

Digital Library

[37]

Cheng Zhang, Aman Parnami, Caleb Southern, Edison Thomaz, Gabriel Reyes, Rosa Arriaga, and Gregory D Abowd. 2013. BackTap: robust four-point tapping on the back of an off-the-shelf smartphone. In Proceedings of ACM UIST.

Digital Library

[38]

Maotian Zhang, Panlong Yang, Chang Tian, Lei Shi, Shaojie Tang, and Fu Xiao. 2015. Soundwrite: Text input on surfaces through mobile acoustic sensing. In Proceedings of ACM SmartObjects.

Digital Library

Cited By

Zhao GShen YLi FLiu LCui LWen H(2025)Ui-Ear: On-Face Gesture Recognition Through On-Ear Vibration SensingIEEE Transactions on Mobile Computing10.1109/TMC.2024.348021624:3(1482-1495)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3480216
Sun TZhao YXie WLi JMa YZhang J(2024)EyeGesener: Eye Gesture Listener for Smart Glasses Interaction Using Acoustic SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785418:3(1-28)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678541
Wang SWang XJiang WMiao CCao QWang HSun KXue HSu L(2024)Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic SignalsACM Transactions on Sensor Networks10.1145/367712220:5(1-32)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3677122
Show More Cited By

Index Terms

Watching Your Phone's Back: Gesture Recognition by Sensing Acoustical Structure-borne Propagation
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

EarCase: Sound Source Localization Leveraging Mini Acoustic Structure Equipped Phone Cases for Hearing-challenged People
MobiHoc '23: Proceedings of the Twenty-fourth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing

Sound source localization is vital for daily tasks such as communication or navigating environments. However, millions of adults struggle with hearing impairment, which limits their ability to identify the direction and distance of sound sources. ...
Spartacus: spatially-aware interaction for mobile devices through energy-efficient audio sensing
MobiSys '13: Proceeding of the 11th annual international conference on Mobile systems, applications, and services

Recent developments in ubiquitous computing enable applications that leverage personal mobile devices, such as smartphones, as a means to interact with other devices in their close proximity. In this paper, we propose Spartacus, a mobile system that ...
A ring-shaped interactive device for large remote display and mobile device control
UbiComp '11: Proceedings of the 13th international conference on Ubiquitous computing

In this demonstration, a novel human-computer interaction device is proposed to realize finger touching for large display and mobile device control, without a touchscreen or a touch pad. In this method, interaction commands are input in a same way as ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 5, Issue 2

June 2021

932 pages

EISSN:2474-9567

DOI:10.1145/3472726

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 June 2021

Published in IMWUT Volume 5, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

NSFC A3 Project
PKU-NTU collaboration Project
PKU-Baidu Funded Project
National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

73
Total Citations
View Citations
759
Total Downloads

Downloads (Last 12 months)104
Downloads (Last 6 weeks)8

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhao GShen YLi FLiu LCui LWen H(2025)Ui-Ear: On-Face Gesture Recognition Through On-Ear Vibration SensingIEEE Transactions on Mobile Computing10.1109/TMC.2024.348021624:3(1482-1495)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3480216
Sun TZhao YXie WLi JMa YZhang J(2024)EyeGesener: Eye Gesture Listener for Smart Glasses Interaction Using Acoustic SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785418:3(1-28)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678541
Wang SWang XJiang WMiao CCao QWang HSun KXue HSu L(2024)Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic SignalsACM Transactions on Sensor Networks10.1145/367712220:5(1-32)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3677122
Zhang JWang J(2024)FusionTrack: Towards Accurate Device-free Acoustic Motion Tracking with Signal FusionACM Transactions on Sensor Networks10.1145/365466620:3(1-30)Online publication date: 6-May-2024
https://dl.acm.org/doi/10.1145/3654666
Chen YYu JChen YKong LZhu YChen YOkoshi TKo JLiKamWa R(2024)RFSpy: Eavesdropping on Online Conversations with Out-of-Vocabulary Words by Sensing Metal Coil Vibration of Headsets Leveraging RFIDProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661887(169-182)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661887
Duan DSun ZNi TLi SJia XXu WLi TOkoshi TKo JLiKamWa R(2024)F2Key: Dynamically Converting Your Face into a Private Key Based on COTS Headphones for Reliable Voice InteractionProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661860(127-140)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661860
Wang CFeng YZhong LZhu SZhang CZheng SLiang CWang YHe CYu CShi Y(2024)UbiPhysioProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435528:1(1-27)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643552
Wang ZWang YYang J(2024)EarSlideProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435158:1(1-29)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643515
Ahmad NLeung H(2024)HyperHARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435118:1(1-29)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643511
Xu ZLiu TJiang RHu PGuo ZLiu C(2024)AFaceProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435108:1(1-33)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643510
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents