skip to main content
10.1145/3498361.3538926acmconferencesArticle/Chapter ViewAbstractPublication PagesmobisysConference Proceedingsconference-collections
research-article

m3Track: <u>mm</u>wave-based <u>m</u>ulti-user 3D posture tracking

Published: 27 June 2022 Publication History

Abstract

Nowadays, the market of 3D human posture tracking has extended to a broad range of application scenarios. As current mainstream solutions, vision-based posture tracking systems suffer from privacy leakage concerns and depend on lighting conditions. Towards more privacy-preserving and robust tracking manner, recent works have exploited commodity radio frequency signals to realize 3D human posture tracking. However, these studies cannot handle the case where multiple users are in the same space. In this paper, we present a <u>mm</u>Wave-based <u>m</u>ulti-user 3D posture tracking system, m3Track, which leverages a single commercial off-the-shelf (COTS) mmWave radar to track multiple users' postures simultaneously as they move, walk, or sit. Based on the sensing signals from a mmWave radar in multi-user scenarios, m3Track first separates all the users on mmWave signals. Then, m3Track extracts shape and motion features of each user, and reconstructs 3D human posture for each user through a designed deep learning model. Furthermore. m3Track maps the reconstructed 3D postures of all users into 3D space, and tracks users' positions through a coordinate-corrected tracking method, realizing practical multi-user 3D posture tracking with a COTS mmWave radar. Experiments conducted in real-world multi-user scenarios validate the accuracy and robustness of m3Track on multi-user 3D posture tracking.

References

[1]
Bernard Boulay, François Bremond, and Monique Thonnat. 2003. Human posture recognition in video sequence. In Proc. IEEE VS-PETS '03.
[2]
Han Cui and Naim Dahnoun. 2021. High Precision Human Detection and Tracking Using Millimeter-Wave Radars. IEEE Aerospace and Electronic Systems Magazine 36, 1 (2021), 22--32.
[3]
Ross Girshick. 2015. Fast r-cnn. In Proc. IEEE ICCV '15. 1440--1448.
[4]
Tianbo Gu, Zheng Fang, Zhicheng Yang, Pengfei Hu, and Prasant Mohapatra. 2019. mmSense: Multi-Person Detection and Identification via mmWave Sensing. In Proc. ACM mmNets@MobiCom '19. Los Cabos, Mexico, 45--50.
[5]
Junfeng Guan, Sohrab Madani, Suraj Jog, Saurabh Gupta, and Haitham Hassanieh. 2020. Through Fog High-Resolution Imaging Using Millimeter Wave Radar. In Proc. IEEE/CVf CVPR '20. WA, USA, 11461--11470.
[6]
Emanuël Anco Peter Habets, Jacob Benesty, Israel Cohen, Sharon Gannot, and Jacek Dmochowski. 2009. New insights into the MVDR beamformer in room acoustics. IEEE Transactions on Audio, Speech, and Language Processing 18, 1 (2009), 158--170.
[7]
Texas Instruments. 2021. AWR1443 Single-chip 76-GHz to 81-GHz automotive radar sensor integrating MCU and hardware accelerator. [Online]. Available: https://www.ti.com/product/AWR1443.
[8]
Texas Instruments. 2021. DCA1000EVM Real-time data-capture adapter for radar sensing evaluation module. [Online]. Available: https://www.ti.com/tool/DCA1000EVM.
[9]
Chengkun Jiang, Junchen Guo, Yuan He, Meng Jin, Shuai Li, and Yunhao Liu. 2020. mmVib: micrometer-level vibration measurement with mmwave radar. In Proc. ACM MobiCom '20. London, United Kingdom, 45:1--45:13.
[10]
Wenjun Jiang, Hongfei Xue, Chenglin Miao, Shiyang Wang, Sen Lin, Chong Tian, Srinivasan Murali, Haochen Hu, Zhi Sun, and Lu Su. 2020. Towards 3D human pose construction using wifi. In Proc. ACM MobiCom '20. London, United Kingdom, 23:1--23:14.
[11]
Wenchao Jiang and Zhaozheng Yin. 2015. Human activity recognition using wearable sensors by deep convolutional neural networks. In Proc. ACM MM '15. Brisbane Australia, 1307--1310.
[12]
Huining Li, Chenhan Xu, Aditya Singh Rathore, Zhengxiong Li, Hanbin Zhang, Chen Song, Kun Wang, Lu Su, Feng Lin, Kui Ren, and Wenyao Xu. 2020. Vocal-Print: exploring a resilient and secure voice authentication via mmWave biometric interrogation. In Proc. ACM SenSys '20. Japan, 312--325.
[13]
Xiang Li, Shengjie Li, Daqing Zhang, Jie Xiong, Yasha Wang, and Hong Mei. 2016. Dynamic-music: accurate device-free indoor localization. In Proc. ACM UbiComp '16. Heidelberg, Germany, 196--207.
[14]
Aristidis Likas, Nikos Vlassis, and Jakob J Verbeek. 2003. The global k-means clustering algorithm. Pattern recognition 36, 2 (2003), 451--461.
[15]
Haipeng Liu, Yuheng Wang, Anfu Zhou, Hanyue He, Wei Wang, Kunpeng Wang, Peilin Pan, Yixuan Lu, Liang Liu, and Huadong Ma. 2020. Real-time Arm Gesture Recognition in Smart Home Scenarios via Millimeter Wave Sensing. Proc. ACM IMWUT 4, 4 (2020), 140:1--140:28.
[16]
Microsoft. 2021. Kinect for Windows. [Online]. Available: https://developer.microsoft.com/en-us/windows/kinect/.
[17]
Kun Qian, Chenshu Wu, Zheng Yang, Yunhao Liu, and Kyle Jamieson. 2017. Widar: Decimeter-level passive tracking via velocity monitoring with commodity Wi-Fi. In Proc. ACM MobiHoc '17. Chennai, India, 6.
[18]
Kun Qian, Chenshu Wu, Yi Zhang, Guidong Zhang, Zheng Yang, and Yunhao Liu. 2018. Widar2.0: Passive Human Tracking with a Single Wi-Fi Link. In Proc. ACM Mobisys '18. Munich, Germany, 350--361.
[19]
Muhannad Quwaider and Subir Biswas. 2008. Body posture identification using hidden Markov model with a wearable sensor network. Bodynets 8 (2008), 1--8.
[20]
Zhou Ren, Junsong Yuan, Jingjing Meng, and Zhengyou Zhang. 2013. Robust part-based hand gesture recognition using kinect sensor. IEEE transactions on multimedia 15, 5 (2013), 1110--1120.
[21]
Maria Isabel Ribeiro. 2004. Kalman and extended kalman filters: Concept, derivation and properties. Institute for Systems and Robotics 43 (2004), 46.
[22]
Hermann Rohling. 1983. Radar CFAR thresholding in clutter and multiple target situations. IEEE transactions on aerospace and electronic systems 4 (1983), 608--621.
[23]
Arindam Sengupta, Feng Jin, and Siyang Cao. 2020. NLP based Skeletal Pose Estimation using mmWave Radar Point-Cloud: A Simulation Approach. In Proc. IEEE RadarConf '20. Florence, USA, 1--6.
[24]
Arindam Sengupta, Feng Jin, Renyuan Zhang, and Siyang Cao. 2019. mmPose: Real-Time Human Skeletal Posture Estimation using mmWave Radars and CNNs. CoRR abs/1911.09592 (2019).
[25]
Agnieszka Sorokowska, Piotr Sorokowski, Peter Hilpert, Katarzyna Cantarero, Tomasz Frackowiak, Khodabakhsh Ahmadi, Ahmad M Alghraibeh, Richmond Aryeetey, Anna Bertoni, Karim Bettache, et al. 2017. Preferred interpersonal distances: a global comparison. Journal of Cross-Cultural Psychology 48, 4 (2017), 577--592.
[26]
Jianwu Wang, Zhichuan Huang, Wenbin Zhang, Ankita Patii, Ketan Patil, Ting Zhu, Eric J Shiroma, Mitchell A Schepps, and Tamara B Harris. 2016. Wearable sensor based human posture recognition. In Proc. IEEE Big Data '16. IEEE, Washington DC, USA, 3432--3438.
[27]
Chenshu Wu, Feng Zhang, Beibei Wang, and K. J. Ray Liu. 2020. mmTrack: Passive Multi-Person Localization Using Commodity Millimeter Wave Radio. In Proc. IEEE INEOCOM '20. IEEE, Toronto, ON, Canada, 2400--2409.
[28]
Jiang Xiao, Kaishun Wu, Youwen Yi, Lu Wang, and Lionel M Ni. 2013. Pilot: Passive device-free indoor localization using channel state information. In Proc. IEEE ICDCS '13. IEEE, Philadelphia, Pennsylvania, USA, 236--245.
[29]
Yaxiong Xie, Jie Xiong, Mo Li, and Kyle Jamieson. 2019. mD-Track: Leveraging multi-dimensionality for passive indoor Wi-Fi tracking. In Proc. ACM MOBICOM '19. Los Cabos, Mexico, 1--16.
[30]
SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. 2015. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Proc. NeurIPS '15. Montreal, Quebec, Canada, 802--810.
[31]
Chenhan Xu, Zhengxiong Li, Hanbin Zhang, Aditya Singh Rathore, Huining Li, Chen Song, Kun Wang, and Wenyao Xu. 2019. WaveEar: Exploring a mmWave-based Noise-resistant Speech Sensing for Voice-User Interface. In Proc. ACM MobiSys '19. Seoul, Republic of Korea, 14--26.
[32]
Hongfei Xue, Yan Ju, Chenglin Miao, Yijiang Wang, Shiyang Wang, Aidong Zhang, and Lu Su. 2021. mmMesh: towards 3D real-time dynamic human mesh construction using millimeter-wave. In Proc. ACM MobiSys '21. Wisconsin, USA, 269--282.
[33]
Zhicheng Yang, Parth H Pathak, Yunze Zeng, Xixi Liran, and Prasant Mohapatra. 2016. Monitoring vital signs using millimeter wave. In Proc. ACM MobiHoc '16. Paderborn, Germany, 211--220.
[34]
Zhicheng Yang, Parth H Pathak, Yunze Zeng, Xixi Liran, and Prasant Mohapatra. 2017. Vital sign and sleep monitoring using millimeter wave. ACM Transactions on Sensor Networks 13, 2 (2017), 1--32.
[35]
Renyuan Zhang and Siyang Cao. 2018. Real-time human motion behavior detection via CNN using mmWave radar. IEEE Sensors Letters 3, 2 (2018), 1--4.
[36]
Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. 2018. Through-wall human pose estimation using radio signals. In Proc. IEEE CVPR '18. Salt Lake City, UT, USA, 7356--7365.
[37]
Mingmin Zhao, Yingcheng Liu, Aniruddh Raghu, Tianhong Li, Hang Zhao, Antonio Torralba, and Dina Katabi. 2019. Through-wall human mesh recovery using radio signals. In Proc. IEEE/CVF ICCV '19. Seoul, Korea (South), 10113--10122.
[38]
Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, Tianhong Li, Rumen Hristov, Zachary Kabelac, Dina Katabi, and Antonio Torralba. 2018. RF-based 3D skeletons. In Proc. ACM SIGCOMM '18. Budapest, Hungary, 267--281.
[39]
Peijun Zhao, Chris Xiaoxuan Lu, Jianan Wang, Changhao Chen, Wei Wang, Niki Trigoni, and Andrew Markham. 2019. mID: Tracking and Identifying People with Millimeter Wave Radar. In Proc. IEEE DCOSS '19. Santorini, Greece, 33--40.

Cited By

View all
  • (2025)Multi-subject human activities: A survey of recognition and evaluation methods based on a formal frameworkExpert Systems with Applications10.1016/j.eswa.2024.126178267(126178)Online publication date: Apr-2025
  • (2024)3D Bounding Box Estimation Based on COTS mmWave Radar via Moving ScanningProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997588:4(1-27)Online publication date: 21-Nov-2024
  • (2024)Artificial Intelligence of Things: A SurveyACM Transactions on Sensor Networks10.1145/369063921:1(1-75)Online publication date: 30-Aug-2024
  • Show More Cited By

Index Terms

  1. m3Track: mmwave-based multi-user 3D posture tracking

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MobiSys '22: Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services
    June 2022
    668 pages
    ISBN:9781450391856
    DOI:10.1145/3498361
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 June 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. 3D posture tracking
    2. mmWave
    3. multi-user scenarios
    4. radars

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    MobiSys '22

    Acceptance Rates

    Overall Acceptance Rate 274 of 1,679 submissions, 16%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)371
    • Downloads (Last 6 weeks)35
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Multi-subject human activities: A survey of recognition and evaluation methods based on a formal frameworkExpert Systems with Applications10.1016/j.eswa.2024.126178267(126178)Online publication date: Apr-2025
    • (2024)3D Bounding Box Estimation Based on COTS mmWave Radar via Moving ScanningProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997588:4(1-27)Online publication date: 21-Nov-2024
    • (2024)Artificial Intelligence of Things: A SurveyACM Transactions on Sensor Networks10.1145/369063921:1(1-75)Online publication date: 30-Aug-2024
    • (2024)View-agnostic Human Exercise Cataloging with Single MmWave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785128:3(1-23)Online publication date: 9-Sep-2024
    • (2024)Mission: mmWave Radar Person Identification with RGB CamerasProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699340(309-321)Online publication date: 4-Nov-2024
    • (2024)Behaviors Speak More: Achieving User Authentication Leveraging Facial Activities via mmWave SensingProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699330(169-183)Online publication date: 4-Nov-2024
    • (2024)SuperSight: Sub-cm NLOS Localization for mmWave BackscatterProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661857(278-291)Online publication date: 3-Jun-2024
    • (2024)Demo: UWB localization and Tracking for XR ApplicationsProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661840(604-605)Online publication date: 3-Jun-2024
    • (2024)Demo: Real-time mmWave Radar Human Sensing TestbedProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3698860(1787-1789)Online publication date: 4-Dec-2024
    • (2024)Demo: Enabling Visual Recognition at Radio FrequencyProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3698859(1784-1786)Online publication date: 4-Dec-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media