research-article

XGest: Enabling Cross-Label Gesture Recognition with RF Signals

Authors:

Li ZhangAuthors Info & Claims

ACM Transactions on Sensor Networks (TOSN), Volume 17, Issue 4

Article No.: 37, Pages 1 - 23

https://doi.org/10.1145/3458750

Published: 03 September 2021 Publication History

Abstract

Extensive efforts have been devoted to human gesture recognition with radio frequency (RF) signals. However, their performance degrades when applied to novel gesture classes that have never been seen in the training set. To handle unseen gestures, extra efforts are inevitable in terms of data collection and model retraining. In this article, we present XGest, a cross-label gesture recognition system that can accurately recognize gestures outside of the predefined gesture set with zero extra training effort. The key insight of XGest is to build a knowledge transfer framework between different gesture datasets. Specifically, we design a novel deep neural network to embed gestures into a high-dimensional Euclidean space. Several techniques are designed to tackle the spatial resolution limits imposed by RF hardware and the specular reflection effect of RF signals in this model. We implement XGest on a commodity mmWave device, and extensive experiments have demonstrated the significant recognition performance.

References

[1]

Microsoft. 2020. Kinect for Windows. Retrieved June 5, 2021 from https://developer.microsoft.com/en-us/windows/kinect/.

[2]

Poser. 2020. Poser Software. Retrieved June 5, 2021 from https://www.posersoftware.com/.

[3]

Texas Instruments. 2020. DCA1000EVM. Retrieved June 5, 2021 from http://www.ti.com/tool/DCA1000EVM.

[4]

Texas Instruments. 2020. IWR Sensor. Retrieved June 5, 2021 from http://www.ti.com/sensors/mmwave/iwr/products.html.

[5]

Heba Abdelnasser, Moustafa Youssef, and Khaled A. Harras. 2015. WiGest: A ubiquitous wifi-based gesture recognition system. In Proceedings of IEEE INFOCOM.

[6]

Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015. Capturing the human figure through a wall. ACM Transactions on Graphics 34, 6 (Nov. 2015), 219.

Digital Library

[7]

D. E. Barrick and Environmental Research Laboratories (U.S.). 1973. FM/CW Radar Signals and Digital Processing. No. v. 55 in FM/CW Radar Signals and Digital Processing. Environmental Research Laboratories.

[8]

Nancy Berryman Reese and William D. Bandy. 2016. Joint Range of Motion and Muscle Length Testing (3rd ed.).Elsevier Health Sciences.

[9]

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555.

[10]

Coupete Eva, Moutarde Fabien, and Manitsaris Sotiris. 2015. Gesture recognition using a depth camera for human robot collaboration on assembly line. Procedia Manufacturing 3 (2015), 518–525.

[11]

Wenjun Jiang, Chenglin Miao, Fenglong Ma, Shuochao Yao, Yaqing Wang, Ye Yuan, Hongfei Xue, et al. 2018. Towards environment independent device free human activity recognition. In Proceedings of ACM MobiCom.

Digital Library

[12]

Wenjun Jiang, Hongfei Xue, Chenglin Miao, Shiyang Wang, Sen Lin, Chong Tian, Srinivasan Murali, Haochen Hu, Zhi Sun, and Lu Su. 2020. Towards 3D human pose construction using wifi. In Proceedings of ACM MobiCom.

Digital Library

[13]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017. ImageNet classification with deep convolutional neural networks. Communications of the ACM 60, 6 (2017), 84–90.

Digital Library

[14]

Tianhong Li, Lijie Fan, Mingmin Zhao, Yingcheng Liu, and Dina Katabi. 2019. Making the invisible visible: Action recognition through walls and occlusions. In Proceedings of IEEE ICCV. 872–881.

[15]

Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics 35, 4 (July 2016), Article 142, 19 pages.

Digital Library

[16]

Yang Liu, Zhenjiang Li, Zhidan Liu, and Kaishun Wu. 2019. Real-Time arm skeleton tracking and gesture inference tolerant to missing wearable sensors. In Proceedings of ACM MobiSys.

Digital Library

[17]

Yu Liu, Yuheng Wang, Haipeng Liu, Anfu Zhou, Jianhua Liu, and Ning Yang. 2020. Long-range gesture recognition using millimeter wave radar. arxiv:cs.HC/2002.02591.

[18]

Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic. 2019. The jester dataset: A large-scale video dataset of human gestures. In Proceedings of IEEE ICCV. 2874–2882.

[19]

Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. 2017. VNect: Real-time 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics 36 (July 2017), Article 44, 1–14.

[20]

Kai Niu, Fusang Zhang, Jie Xiong, Xiang Li, Enze Yi, and Daqing Zhang. 2018. Boosting fine-grained activity sensing by embracing wireless multipath effects. In Proceedings of ACM CoNext.

Digital Library

[21]

Joel C. Perry and Jacob Rosen. 2006. Design of a 7 degree-of-freedom upper-limb powered exoskeleton. In Proceedings of IEEE BioRob.

[22]

Kun Qian, Chenshu Wu, Fu Xiao, Yue Zheng, Yi Zhang, Zheng Yang, and Yunhao Liu. 2018. Acousticcardiogram: Monitoring heartbeats using acoustic signals on smart devices. In Proceedings of IEEE INFOCOM.

Digital Library

[23]

Kun Qian, Chenshu Wu, Zheng Yang, Yunhao Liu, and Kyle Jamieson. 2017. Widar: Decimeter-level passive tracking via velocity monitoring with commodity wi-fi. In Proceedings of ACM MobiHoc.

Digital Library

[24]

Kun Qian, Chenshu Wu, Yi Zhang, Guidong Zhang, Zheng Yang, and Yunhao Liu. 2018. Widar2.0: Passive human tracking with a single wi-fi link. Proceedings of ACM MobiSys.

Digital Library

[25]

Kun Qian, Chenshu Wu, Zimu Zhou, Yue Zheng, Zheng Yang, and Yunhao Liu. 2017. Inferring motion direction using commodity wi-fi for interactive exergames. In Proceedings of ACM CHI.

Digital Library

[26]

Markus Scholz, Till Riedel, Mario Hock, and Michael Beigl. 2013. Device-free and device-bound activity recognition using radio signal strength. In Proceedings of ACM AH.

Digital Library

[27]

Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I am a smartwatch and I can track my user’s arm. In Proceedings of ACM MobiSys.

Digital Library

[28]

Stephan Sigg, Shuyu Shi, Felix Buesching, Yusheng Ji, and Lars Wolf. 2013. Leveraging RF-Channel fluctuation for activity recognition: Active and passive systems, continuous and RSSI-based signal features. In Proceedings of ACM MoMM.

Digital Library

[29]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS.

Digital Library

[30]

Raghav H. Venkatnarayan, Griffin Page, and Muhammad Shahzad. 2018. Multi-user gesture recognition using wifi. In Proceedings of ACM MobiSys.

Digital Library

[31]

Aditya Virmani and Muhammad Shahzad. 2017. Position and orientation agnostic gesture recognition using wifi. In Proceedings of ACM MobiSys.

Digital Library

[32]

Chuyu Wang, Jian Liu, Yingying Chen, Hongbo Liu, Lei Xie, Wei Wang, Bingbing He, and Sanglu Lu. 2018. Multi-touch in the air: Device-free finger tracking and gesture recognition via COTS RFID. In Proceedings of IEEE INFOCOM.

Digital Library

[33]

He Wang, Ted Tsung-Te Lai, and Romit Roy Choudhury. 2015. MoLe: Motion leaks through smartwatch sensors. In Proceedings of ACM MobiCom.

Digital Library

[34]

Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: Virtual touch screen in the air using RF signals. In Proceedings of ACM SIGCOMM.

Digital Library

[35]

Jie Wang, Xia Zhang, Qinhua Gao, Hao Yue, and Hongyu Wang. 2017. Device-free wireless localization and activity recognition: A deep learning approach. IEEE Transactions on Vehicular Technology 66 (2017), 6258–6267.

[36]

Qinyi Wang, Yexin Zhang, Junsng Yuan, and Yihong Lu. 2019. Space-time event clouds for gesture recognition: From RGB cameras to event cameras. In Proceedings of IEEE WACV.

[37]

Wei Wang, Alex X. Liu, Muhammad Shahzad, Kang Ling, and Sanglu Lu. 2015. Understanding and modeling of wifi signal based human activity recognition. In Proceedings of ACM MobiCom.

Digital Library

[38]

Yan Wang, Jian Liu, Yingying Chen, Marco Gruteser, Jie Yang, and Hongbo Liu. 2014. E-eyes: Device-free location-oriented activity identification using fine-grained wifi signatures. In Proceedings of ACM MobiCom.

Digital Library

[39]

Teng Wei and Xinyu Zhang. 2015. MTrack: High-precision passive tracking using millimeter wave radios. In Proceedings of ACM MobiCom.

Digital Library

[40]

Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. ISkin: Flexible, stretchable and visually customizable on-body touch sensors for mobile computing. In Proceedings of ACM CHI.

Digital Library

[41]

Chenshu Wu, Feng Zhang, Yusen Fan, and K. J. Ray Liu. 2019. RF-based inertial measurement. In Proceedings of ACM SIGCOMM. 117–129.

[42]

Chao Xu, Parth H. Pathak, and Prasant Mohapatra. 2015. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. In Proceedings of ACM HotMobile.

Digital Library

[43]

Zheng Yang, Zimu Zhou, and Yunhao Liu. 2013. From RSSI to CSI: Indoor localization via channel response. ACM Computing Surveys 46, 2 (Nov. 2013), Article 25, 32 pages.

Digital Library

[44]

Yafeng Yin, Lei Xie, Tao Gu, Yijia Lu, and Sanglu Lu. 2019. AirContour: Building contour-based model for in-air writing gesture recognition. ACM Transactions on Sensor Networks 15, 4 (Oct. 2019), Article 44, 25 pages.

Digital Library

[45]

Jie Zhang, Zhanyong Tang, Meng Li, Dingyi Fang, Petteri Tapio Nurmi, and Zheng Wang. 2018. CrossSense: Towards cross-site and large-scale wifi sensing. In Proceedings of ACM MobiCom.

Digital Library

[46]

Yi Zhang, Yue Zheng, Guidong Zhang, Kun Qian, Chen Qian, and Zheng Yang. 2020. GaitID: Robust wi-fi based gait recognition. In Proceedings of Springer WASA.

[47]

Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, Tianhong Li, Rumen Hristov, Zachary Kabelac, Dina Katabi, and Antonio Torralba. 2018. RF-Based 3D skeletons. In Proceedings of ACM SIGCOMM.

Digital Library

[48]

Yue Zheng, Yi Zhang, Kun Qian, Guidong Zhang, Yunhao Liu, Chenshu Wu, and Zheng Yang. 2019. Zero-effort cross-domain gesture recognition with wi-fi. In Proceedings of ACM MobiSys. 313–325.

Digital Library

Cited By

Liu YRen YLiu MLi HGuo HMiao XHu XChen HMa XWon Y(2024)Optimizing file systems on heterogeneous memory by integrating DRAM cache with virtual memory managementProceedings of the 22nd USENIX Conference on File and Storage Technologies10.5555/3650697.3650702(71-88)Online publication date: 27-Feb-2024
https://dl.acm.org/doi/10.5555/3650697.3650702
Li LBai WHan C(2024)Multiscale Feature Fusion for Gesture Recognition Using Commodity Millimeter-Wave RadarComputers, Materials & Continua10.32604/cmc.2024.05607381:1(1613-1640)Online publication date: 2024
https://doi.org/10.32604/cmc.2024.056073
Niaz FZhang JZheng YKhalid MNiaz A(2024)mm-CUR: A Novel Ubiquitous, Contact-free, and Location-aware Counterfeit Currency Detection in Bundles Using Millimeter-Wave SensorACM Transactions on Sensor Networks10.1145/369497020:6(1-26)Online publication date: 5-Sep-2024
https://dl.acm.org/doi/10.1145/3694970
Show More Cited By

Index Terms

XGest: Enabling Cross-Label Gesture Recognition with RF Signals
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools

Recommendations

Multi-scenario gesture recognition using Kinect
CGAMES '12: Proceedings of the 2012 17th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games (CGAMES)

Hand gesture recognition (HGR) is an important research topic because some situations require silent communication with sign languages. Computational HGR systems assist silent communication, and help people learn a sign language. In this article, a ...
Enabling Finger-Gesture Interaction with Kinect
VINCI '15: Proceedings of the 8th International Symposium on Visual Information Communication and Interaction

A large number of tracking and gesture recognition algorithms and technologies have been developed in the field of human-computer interactions thanks to the introduction of cameras with depth sensors such as Microsoft's Kinect. Most of the techniques ...
Recent methods and databases in vision-based hand gesture recognition

The paper surveys RGB and RGB-D sensors based hand gesture recognition methods.Dynamic as well as static gesture (posture/pose) recognition methods are reviewed.Qualitative as well as quantitative comparison of algorithms is provided.Twenty-six publicly ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Sensor Networks

ACM Transactions on Sensor Networks Volume 17, Issue 4

November 2021

403 pages

ISSN:1550-4859

EISSN:1550-4867

DOI:10.1145/3472298

Editor:
Yunhao Liu
Tsinghua University, China

Issue’s Table of Contents

Copyright © 2021 Association for Computing Machinery.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

ACM Journals for the Design of Smart and Connected Systems

Publication History

Published: 03 September 2021

Accepted: 01 March 2021

Revised: 01 February 2021

Received: 01 December 2020

Published in TOSN Volume 17, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Key R&D Program of China
NSFC

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
535
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu YRen YLiu MLi HGuo HMiao XHu XChen HMa XWon Y(2024)Optimizing file systems on heterogeneous memory by integrating DRAM cache with virtual memory managementProceedings of the 22nd USENIX Conference on File and Storage Technologies10.5555/3650697.3650702(71-88)Online publication date: 27-Feb-2024
https://dl.acm.org/doi/10.5555/3650697.3650702
Li LBai WHan C(2024)Multiscale Feature Fusion for Gesture Recognition Using Commodity Millimeter-Wave RadarComputers, Materials & Continua10.32604/cmc.2024.05607381:1(1613-1640)Online publication date: 2024
https://doi.org/10.32604/cmc.2024.056073
Niaz FZhang JZheng YKhalid MNiaz A(2024)mm-CUR: A Novel Ubiquitous, Contact-free, and Location-aware Counterfeit Currency Detection in Bundles Using Millimeter-Wave SensorACM Transactions on Sensor Networks10.1145/369497020:6(1-26)Online publication date: 5-Sep-2024
https://dl.acm.org/doi/10.1145/3694970
Sheng BHan RXiao FGuo ZGui L(2024)MetaFormerProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435508:1(1-27)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643550
Wang FLv YZhu MDing HHan J(2024)XRF55Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435438:1(1-34)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643543
Liao PWang XAn LMao SZhao TYang C(2024)TFSemantic: A Time–Frequency Semantic GAN Framework for Imbalanced Classification Using Radio SignalsACM Transactions on Sensor Networks10.1145/361409620:4(1-22)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3614096
Sheng BHan RCai HXiao FGui LGuo Z(2024)CDFi: Cross-Domain Action Recognition Using WiFi SignalsIEEE Transactions on Mobile Computing10.1109/TMC.2023.334893923:8(8463-8477)Online publication date: Aug-2024
https://doi.org/10.1109/TMC.2023.3348939
Cai MJiang XShen JYe B(2024)SplitDB: Closing the Performance Gap for LSM-Tree-Based Key-Value StoresIEEE Transactions on Computers10.1109/TC.2023.332698273:1(206-220)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TC.2023.3326982
Liu YLi YCheng WWang WYang J(2023)A Reputation-Based Collaborative User Recruitment Algorithm in Edge-Aided Mobile CrowdsensingApplied Sciences10.3390/app1310604013:10(6040)Online publication date: 14-May-2023
https://doi.org/10.3390/app13106040
Yao TLi YLi YZhu YWang GYue J(2023)Cross-modal Semantically Augmented Network for Image-text MatchingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363135620:4(1-18)Online publication date: 11-Dec-2023
https://dl.acm.org/doi/10.1145/3631356
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents