Haipeng Liu
Skip Abstract Section
Abstract
Gesture recognition using millimeter-wave radios facilitates natural human-computer interactions, but existing works require a consistent environment, i.e., the neural networks for recognition are trained and tested for the same users and at some fixed positions. In this case, their performance will decrease rapidly when they enter into a new environment. To make the model applicable in different environments, a straightforward approach is to collect and re-train the model for the gesture samples on every possible position upon each new user. However, it may ask the users to spend unacceptable time to accomplish such adaptation, which makes it difficult to be widely used in practice. In this paper, we first collect an abundant mmWave gesture dataset containing 59,280 samples as a benchmark to investigate the impact of the environment changes quantitatively. Then we propose a novel transfer-learning approach called mTransSee, which can serve the gestures in practice using pre-learned experience by least adaptation, i.e., retraining using only 8 samples per gesture for the same accuracy. mTransSee reduces dozens of workloads for the environment adaptation. We implement mTransSee on a commodity mmWave sensor and make a user study to compare the advance of mTransSee over the state-of-the-art solution in terms of user experience during adaptation.
- 2020. Google Pixel 4 in Nov. 2019. https://store.google.com/product/pixel_4.Google Scholar
- 2022. IWR6843: Single-chip 60-GHz to 64-GHz mmWave sensor. https://www.ti.com/product/IWR6843.Google Scholar
- 2022. Project soli mmWave radar. https://atap.google.com/soli/.Google Scholar
- Paszke Adam. 2019. PyTorch: An open-source deep learning platform. https://github.com/pytorch/pytorch.Google Scholar
- Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 2014. 3D tracking via body radio reflections. In NSDI.Google Scholar
- Anonymity. 2021. Anonymous dataset. https://github.com/mmTransGes/mTransSee_Dataset.Google Scholar
- Hossein Azizpour, A. Razavian, J. Sullivan, A. Maki, and S. Carlsson. 2016. Factors of Transferability for a Generic ConvNet Representation. TPAMI (2016).Google Scholar
- Dane Bertram. 2007. Likert scales. Retrieved November 2 (2007), 2013.Google Scholar
- Ling Chen, Yi Zhang, and Liangying Peng. 2020. METIER: A Deep Multi-Task Learning Based Activity and User Recognition Model Using Wearable Sensors. IMWUT, Article 5 (March 2020), 18 pages.Google ScholarDigital Library
- Martin Ester, Hans-Peter Kriegel, et al. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise.. In Kdd.Google Scholar
- Li Fei-Fei, R. Fergus, and P. Perona. 2006. One-shot learning of object categories. IEEE TPAMI (2006).Google Scholar
- Borja Gonzalez-Valdes, Yuri Álvarez, Yolanda Rodriguez-Vaqueiro, Ana Arboleya-Arboleya, Antonio García-Pino, Carey M. Rappaport, Fernando Las-Heras, and Jose A. Martinez-Lorenzo. 2016. Millimeter Wave Imaging Architecture for On-The-Move Whole Body Imaging. IEEE Transactions on Antennas and Propagation (2016).Google Scholar
- Tianbo Gu, Zheng Fang, Zhicheng Yang, Pengfei Hu, and Prasant Mohapatra. 2019. MmSense: Multi-Person Detection and Identification via MmWave Sensing. In mmNets.Google Scholar
- Eiji Hayashi, Jaime Lien, Nicholas Gillian, Leonardo Giusti, Dave Weber, Jin Yamanaka, Lauren Bedal, and Ivan Poupyrev. 2021. RadarNet: Efficient Gesture Recognition Technique Utilizing a Miniature Radar Sensor. In ACM CHI.Google Scholar
- W. He, K. Wu, Y. Zou, and Z. Ming. 2015. WiG: WiFi-Based Gesture Recognition System. In 2015 24th International Conference on Computer Communication and Networks (ICCCN). 1--7.Google Scholar
- Texas Instruments. 2017. Mmwave sdk user guide.Google Scholar
- Cesar Iovescu and Sandeep Rao. 2017. The fundamentals of millimeter wave sensors. Texas Instruments (2017).Google Scholar
- Rong Jin, Shijun Wang, and Yang Zhou. 2009. Regularized distance metric learning: Theory and algorithm. NIPS (2009).Google Scholar
- F. Lécué, Jiaoyan Chen, J. Pan, and Huajun Chen. 2019. Augmenting Transfer Learning with Semantic Reasoning. In IJCAI.Google Scholar
- Hong et al. Li. 2016. WiFinger: talk to your smart devices with finger-grained gesture. (2016), 250--261.Google Scholar
- Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ToG (2016).Google ScholarDigital Library
- Haipeng Liu, Yuheng Wang, et al. 2020. Real-time Arm Gesture Recognition in Smart Home Scenarios via Millimeter Wave Sensing. ACM IMWUT (2020).Google Scholar
- Haipeng Liu, Anfu Zhou, Zihe Dong, Yuyang Sun, Jiahe Zhang, Liang Liu, Huadong Ma, Jianhua Liu, and Ning Yang. 2021. M-Gesture: Person-Independent Real-Time In-Air Gesture Recognition Using Commodity Millimeter Wave Radar. IEEE Internet of Things Journal (2021).Google Scholar
- X. Liu, K. Liu, X. Li, Jinsong Su, Y. Ge, Bin Wang, and Jiebo Luo. 2020. An Iterative Multi-Source Mutual Knowledge Transfer Framework for Machine Reading Comprehension. In IJCAI.Google Scholar
- Yiqin Lu, Bingjian Huang, Chun Yu, Guahong Liu, and Yuanchun Shi. 2020. Designing and Evaluating Hand-to-Hand Gestures with Dual Commodity Wrist-Worn Devices. IMWUT, Article 20 (March 2020), 27 pages.Google Scholar
- Maaten, Hinton Laurens, and Geoffrey. 2008. Visualizing Data using t-SNE. JMLR (2008).Google Scholar
- Wenguang Mao, Mei Wang, Wei Sun, Lili Qiu, Swadhin Pradhan, and Yi-Chao Chen. 2019. RNN-Based Room Scale Hand Motion Tracking (MobiCom '19). Association for Computing Machinery, New York, NY, USA, Article 38, 16 pages.Google Scholar
- Yong Niu, Yong Li, Depeng Jin, Li Su, and Athanasios V Vasilakos. 2015. A survey of millimeter wave communications (mmWave) for 5G: opportunities and challenges. Wireless Networks (2015).Google Scholar
- Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash. 2018. Boosting Self-Supervised Learning via Knowledge Transfer. In CVPR.Google Scholar
- Mark Palatucci, Dean Pomerleau, Geoffrey Hinton, and Tom Mitchell. 2009. Zero-shot Learning with Semantic Output Codes. NIPS.Google Scholar
- Sinno Jialin Pan and Qiang Yang. 2010. A Survey on Transfer Learning. IEEE TKDE (2010).Google ScholarDigital Library
- Avishek Patra, Philipp Geuer, Andrea Munari, and Petri Mähönen. 2018. Mm-Wave Radar Based Gesture Recognition: Development and Evaluation of a Low-Power, Low-Complexity System. In ACM mmNets.Google Scholar
- Avishek Patra, Philipp Geuer, Andrea Munari, and Petri Mähönen. 2018. mm-Wave Radar Based Gesture Recognition: Development and Evaluation of a Low-Power, Low-Complexity System. In Proceedings of the 2nd ACM Workshop on Millimeter Wave Networks and Sensing Systems. ACM, 51--56.Google ScholarDigital Library
- Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak N Patel. 2013. Whole-home gesture recognition using wireless signals. (2013), 27--38.Google Scholar
- Valentin Radu and Maximilian Henne. 2019. Vision2Sensor: Knowledge Transfer Across Sensing Modalities Human Activity Recognition. IWMUT (2019).Google ScholarDigital Library
- Hermann Rohling. 1983. Radar CFAR Thresholding in Clutter and Multiple Target Situations. IEEE Trans. Aerospace Electron. Systems (1983).Google ScholarCross Ref
- Panneer Selvam Santhalingam, Al Amin Hosain, Ding Zhang, Parth Pathak, Huzefa Rangwala, and Raja Kushalnagar. 2020. MmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-Wave Signals. ACM IMWUT (2020).Google Scholar
- Muhammad Shahzad and Shaohu Zhang. 2018. Augmenting user identification with WiFi based gesture recognition. IMWUT 2, 3 (2018), 1--27.Google Scholar
- Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, et al. 2015. Accurate, robust, and flexible real-time hand tracking. In ACM CHI.Google Scholar
- Akash Deep Singh, Sandeep Singh Sandha, Luis Garcia, and Mani Srivastava. 2019. RadHAR: Human Activity Recognition from Point Clouds Generated through a Millimeter-Wave Radar. In ACM mmNets.Google Scholar
- Sruthy Skaria, Akram Alhourani, Margaret Lech, and R J Evans. 2019. Hand-Gesture Recognition Using Two-Antenna Doppler Radar With Deep Convolutional Neural Networks. IEEE Sensors Journal (2019).Google ScholarCross Ref
- Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for few-shot learning. NIPS (2017).Google ScholarDigital Library
- Princeton University Stanford Vision Lab, Stanford University. 2019. ImageNet is an image database. http://www.image-net.org/.Google Scholar
- Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. 2018. Learning to Compare: Relation Network for Few-Shot Learning. In CVPR.Google Scholar
- Nima Tajbakhsh, Jae Y. Shin, S. Gurudu, R. T. Hurst, Christopher B. Kendall, M. Gotway, and Jianming Liang. 2016. Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? IEEE Transactions on Medical Imaging (2016).Google Scholar
- Jiayao Tan, Xiaoliang Wang, Cam-Tu Nguyen, and Yu Shi. 2018. SilentKey: A New Authentication Framework through Ultrasonic-Based Lip Reading. IWMUT, Article 36 (March 2018), 18 pages.Google ScholarDigital Library
- Yonglong Tian, Guang-He Lee, Hao He, Chen-Yu Hsu, and Dina Katabi. 2018. RF-Based Fall Monitoring Using Convolutional Neural Networks. IMWUT 2, 3 (2018), 137.Google Scholar
- Raghav H Venkatnarayan, Griffin Page, and Muhammad Shahzad. 2018. Multi-User Gesture Recognition Using WiFi. (2018), 401--413.Google Scholar
- Aditya Virmani and Muhammad Shahzad. 2017. Position and Orientation Agnostic Gesture Recognition Using WiFi. (2017), 252--264.Google Scholar
- Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. 2019. Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics. In CVPR.Google Scholar
- Leye Wang, Xu Geng, Xiaojuan Ma, F. Liu, and Q. Yang. 2019. Cross-City Transfer Learning for Deep Spatio-Temporal Prediction. In IJCAI.Google Scholar
- Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum. In ACM UIST.Google Scholar
- Teng Wei and Xinyu Zhang. 2015. mTrack: High-Precision Passive Tracking Using Millimeter Wave Radios. In MobiCom.Google Scholar
- Piotr Wojtczuk, David Binnie, Alistair Armitage, Tim Chamberlain, and Carsten Giebeler. 2013. A Touchless Passive Infrared Gesture Sensor. In ACM UIST (Adjunct).Google Scholar
- Wanqi Xue and Wei Wang. 2020. One-Shot Image Classification by Learning to Restore Prototypes. AAAI (2020).Google Scholar
- Zhicheng Yang, Parth H Pathak, Yunze Zeng, Xixi Liran, and Prasant Mohapatra. 2016. Monitoring vital signs using millimeter wave. In ACM MobiHoc.Google Scholar
- Zhongjie Yu, L. Chen, Zhongwei Cheng, and Jiebo Luo. 2020. TransMatch: A Transfer-Learning Scheme for Semi-Supervised Few-Shot Learning. CVPR (2020).Google Scholar
- Wang Yuheng, Liu Haipeng, Cui Kening, Zhou Anfu, Li Wensheng, and Ma Huadong. 2021. m-Activity: Accurate and real-time human activity recognition via millimeter wave radar. In IEEE ICASSP.Google Scholar
- M. Zeiler, G. Taylor, and R. Fergus. 2011. Adaptive deconvolutional networks for mid and high level feature learning.. In ICCV.Google Scholar
- Jie Zhang, Zhanyong Tang, Meng Li, Dingyi Fang, Petteri Nurmi, and Zheng Wang. 2018. CrossSense: Towards Cross-Site and Large-Scale WiFi Sensing. (2018), 305--320.Google Scholar
- Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, Tianhong Li, Rumen Hristov, Zachary Kabelac, Dina Katabi, and Antonio Torralba. 2018. RF-based 3D skeletons. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. ACM, 267--281.Google ScholarDigital Library
- Meng Zhen, Fu Song, et al. 2020. Gait Recognition for Co-existing Multiple People Using Millimeter Wave Sensing. AAAI (2020).Google Scholar
- Yue Zheng, Yi Zhang, Kun Qian, Guidong Zhang, Yunhao Liu, Chenshu Wu, and Zheng Yang. 2019. Zero-Effort Cross-Domain Gesture Recognition with Wi-Fi. MobiSys (2019).Google Scholar
- Anfu Zhou, Shaoyuan Yang, Yi Yang, Yuhang Fan, and Huadong Ma. 2019. Autonomous Environment Mapping Using Commodity Millimeter-wave Network Device. (2019), 1126--1134.Google Scholar
- Ziye Zhu, Y. Li, H. Tong, and Yu Wang. 2020. CooBa: Cross-project Bug Localization via Adversarial Transfer Learning. In IJCAI.Google Scholar
Index Terms
- mTransSee: Enabling Environment-Independent mmWave Sensing Based Gesture Recognition via Transfer Learning
Recommendations
Multi-scenario gesture recognition using Kinect
CGAMES '12: Proceedings of the 2012 17th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games (CGAMES)Hand gesture recognition (HGR) is an important research topic because some situations require silent communication with sign languages. Computational HGR systems assist silent communication, and help people learn a sign language. In this article, a ...
Real-time Arm Gesture Recognition in Smart Home Scenarios via Millimeter Wave Sensing
"In air" gesture recognition using millimeter wave (mmWave) radar and its applications in natural human-computer-interaction for smart home has shown its potential. However, the state-of-the-art works still fall short in terms of limited gesture space, ...
3D Freehand Gestural Navigation for Interactive Public Displays
Users increasingly expect more-interactive experiences with public displays for applications including learning, gaming, urban visualization, and planning. However, user interaction with applications on public displays is challenging and often doesn't ...
Comments