Demo: Enabling Visual Recognition at Radio Frequency
Pages 1784 - 1786
Abstract
This demo presents PanoRadar, a novel RF imaging system that brings RF resolution close to that of LiDAR, while providing resilience against conditions challenging for optical signals. Our LiDAR-comparable 3D imaging results enable, for the first time, a variety of visual recognition tasks at radio frequency, including surface normal estimation, semantic segmentation, and object detection. PanoRadar utilizes a rotating single-chip mmWave radar, along with a combination of novel signal processing and machine learning algorithms, to create high-resolution 3D images of the surroundings. Our system accurately estimates robot motion, allowing for coherent imaging through a dense grid of synthetic antennas. It also exploits the high azimuth resolution to enhance elevation resolution using learning-based methods. Furthermore, PanoRadar tackles 3D learning via 2D convolutions and addresses challenges due to the unique characteristics of RF signals. This demonstration illustrates the ability of high-resolution RF imaging of PanoRadar. We present the signal processing results to show the high range and azimuth resolution of the system, and the machine learning results for elevation resolution enhancement. Code, datasets, and demo videos are available on our website.
References
[1]
Fadel Adib, Zachary Kabelac, and Dina Katabi. 2015. Multi-person localization via RF body reflections. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15). 279--292.
[2]
Dan Barnes, Matthew Gadd, Paul Murcutt, Paul Newman, and Ingmar Posner. 2020. The oxford radar robotcar dataset: A radar extension to the oxford robotcar dataset. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 6433--6438.
[3]
Tara Boroushaki, Isaac Perper, Mergen Nachin, Alberto Rodriguez, and Fadel Adib. 2021. RFusion: Robotic Grasping via RF-Visual Sensing and Learning. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems (Coimbra, Portugal) (SenSys '21). Association for Computing Machinery, New York, NY, USA, 192--205.
[4]
Keenan Burnett, David J Yoon, Yuchen Wu, Andrew Zou Li, Haowei Zhang, Shichen Lu, Jingxing Qian, Wei-Kang Tseng, Andrew Lambert, Keith YK Leung, et al. 2022. Boreas: A multi-season autonomous driving dataset. arXiv preprint arXiv:2203.10168 (2022).
[5]
Shiwei Fang and Shahriar Nirjon. 2020. Superrf: Enhanced 3d rf representation using stationary low-cost mmwave radar. In International Conference on Embedded Wireless Systems and Networks (EWSN) ..., Vol. 2020. NIH Public Access, 120.
[6]
Junfeng Guan, Sohrab Madani, Suraj Jog, Saurabh Gupta, and Haitham Hassanieh. 2020. Through fog high-resolution imaging using millimeter wave radar. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11464--11473.
[7]
Maki K Habib and Yvan Baudoin. 2010. Robot-assisted risky intervention, search, rescue and environmental surveillance. International Journal of Advanced Robotic Systems 7, 1 (2010), 10.
[8]
Texas Instruments. 2020. AWR1843BOOST and IWR1843BOOST Single-Chip mmWave Sensing Solution User's Guide (Rev. B). Texas Instruments. https://www.ti.com/lit/pdf/spruim4
[9]
Kiran Joshi, Dinesh Bharadia, Manikanta Kotaru, and Sachin Katti. 2015. Wideo: Fine-grained device-free motion tracing using RF backscatter. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15). 189--204.
[10]
Hao Kong, Xiangyu Xu, Jiadi Yu, Qilin Chen, Chenguang Ma, Yingying Chen, Yi-Chao Chen, and Linghe Kong. 2022. M3Track: <u>mm</U>wave-Based <u>m</U>ulti-User 3D Posture Tracking. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services (Portland, Oregon) (MobiSys '22). Association for Computing Machinery, New York, NY, USA, 491--503.
[11]
Haowen Lai, Gaoxiang Luo, Yifei Liu, and Mingmin Zhao. 2024. Enabling Visual Recognition at Radio Frequency. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. 388--403.
[12]
Chris Xiaoxuan Lu, Stefano Rosa, Peijun Zhao, Bing Wang, Changhao Chen, John A Stankovic, Niki Trigoni, and Andrew Markham. 2020. See through smoke: robust indoor mapping with low-cost mmwave radar. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services. 14--27.
[13]
Chris Xiaoxuan Lu, Muhamad Risqi U Saputra, Peijun Zhao, Yasin Almalioglu, Pedro PB De Gusmao, Changhao Chen, Ke Sun, Niki Trigoni, and Andrew Markham. 2020. milliEgo: single-chip mmWave radar aided egomotion estimation via deep sensor fusion. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems. 109--122.
[14]
Sohrab Madani, Jayden Guan, Waleed Ahmed, Saurabh Gupta, and Haitham Hassanieh. 2022. Radatron: Accurate Detection Using Multi-resolution Cascaded MIMO Radar. In 17th European Conference of Computer Vision (ECCV). Springer, 160--178.
[15]
Akarsh Prabhakara, Tao Jin, Arnav Das, Gantavya Bhatt, Lilly Kumari, Elahe Soltanaghai, Jeff Bilmes, Swarun Kumar, and Anthony Rowe. 2023. High Resolution Point Clouds from mmWave Radar. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 4135--4142.
[16]
Akarsh Prabhakara, Vaibhav Singh, Swarun Kumar, and Anthony Rowe. 2020. Osprey: A MmWave Approach to Tire Wear Sensing. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services (Toronto, Ontario, Canada) (MobiSys '20). Association for Computing Machinery, New York, NY, USA, 28--41.
[17]
Kun Qian, Zhaoyuan He, and Xinyu Zhang. 2020. 3D point cloud generation with millimeter-wave radar. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--23.
[18]
Guy Satat, Matthew Tancik, and Ramesh Raskar. 2018. Towards photography through realistic fog. In 2018 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--10.
[19]
Yue Sun, Zhuoming Huang, Honggang Zhang, Zhi Cao, and Deqiang Xu. 2021. 3DRIMR: 3D reconstruction and imaging via mmWave radar based on deep learning. In 2021 IEEE International Performance, Computing, and Communications Conference (IPCCC). IEEE, 1--8.
[20]
Ju Wang, Jie Xiong, Xiaojiang Chen, Hongbo Jiang, Rajesh Krishna Balan, and Dingyi Fang. 2017. TagScan: Simultaneous target imaging and material identification with commodity RFID devices. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking. 288--300.
[21]
Weiye Xu, Wenfan Song, Jianwei Liu, Yajie Liu, Xin Cui, Yuanqing Zheng, Jinsong Han, Xinhuai Wang, and Kui Ren. 2022. Mask does not matter: Anti-spoofing face authentication using mmWave without on-site registration. In Proceedings of the 28th Annual International Conference on Mobile Computing and Networking. 310--323.
[22]
Hongfei Xue, Qiming Cao, Yan Ju, Haochen Hu, Haoyu Wang, Aidong Zhang, and Lu Su. 2023. M4esh: MmWave-Based 3D Human Mesh Construction for Multiple Subjects. In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems (Boston, Massachusetts) (SenSys '22). Association for Computing Machinery, New York, NY, USA, 391--406.
[23]
Hongfei Xue, Yan Ju, Chenglin Miao, Yijiang Wang, Shiyang Wang, Aidong Zhang, and Lu Su. 2021. MmMesh: Towards 3D Real-Time Dynamic Human Mesh Construction Using Millimeter-Wave. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services (Virtual Event, Wisconsin) (MobiSys '21). Association for Computing Machinery, New York, NY, USA, 269--282.
[24]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.
[25]
Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. 2018. Through-wall human pose estimation using radio signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7356--7365.
[26]
Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, Tianhong Li, Rumen Hristov, Zachary Kabelac, Dina Katabi, and Antonio Torralba. 2018. RF-based 3D skeletons. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. 267--281.
Index Terms
- Demo: Enabling Visual Recognition at Radio Frequency
Recommendations
Enabling Visual Recognition at Radio Frequency
ACM MobiCom '24: Proceedings of the 30th Annual International Conference on Mobile Computing and NetworkingThis paper introduces PanoRadar, a novel RF imaging system that brings RF resolution close to that of LiDAR, while providing resilience against conditions challenging for optical signals. Our LiDAR-comparable 3D imaging results enable, for the first time,...
Push the Limit of Single-Chip mmWave Radar-Based Egomotion Estimation with Moving Objects in FoV
SenSys '23: Proceedings of the 21st ACM Conference on Embedded Networked Sensor SystemsThis paper presents EmoRI, a novel single-chip mmWave radar-based egomotion estimation approach that works in challenging scenarios where moving objects exist in radar's Field of View (FoV). Essentially, estimating a mobile platform's egomotion using an ...
Comments
Information & Contributors
Information
Published In
December 2024
2476 pages
ISBN:9798400704895
DOI:10.1145/3636534
- Chair:
- Weisong Shi,
- Program Chairs:
- Deepak Ganesan,
- Nicholas Lane
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 December 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
ACM MobiCom '24
Sponsor:
ACM MobiCom '24: 30th Annual International Conference on Mobile Computing and Networking
November 18 - 22, 2024
DC, Washington D.C., USA
Acceptance Rates
Overall Acceptance Rate 440 of 2,972 submissions, 15%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 78Total Downloads
- Downloads (Last 12 months)78
- Downloads (Last 6 weeks)67
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in