Xingyu Chen
ABSTRACT
This paper presents RF Genesis (RFGen), a novel and cost-effective method for synthesizing RF sensing data using cross-modal diffusion models, in order to improve the generalization capability of millimeter-wave (mmWave) sensing systems. Traditional machine learning models used in mmWave sensing struggle with limited training datasets. Their performance degrades drastically when confronted with unseen users, environments, sensor configurations, test classes, etc. RFGen mitigates these challenges by using a cross-modal generative framework to synthesize and expand mmWave sensing data. We specifically propose a custom ray tracing simulator to simulate RF propagation and interaction with objects/environments. We then leverage a set of diffusion models to generate massive 3D scenes, and transform the visual scene representation into the corresponding mmWave sensing data, under the direction of application-specific "prompts". Our proposed approach reconciles the physics-based ray tracing with the blackbox diffusion model, leading to accurate, scalable, and explainable vision-to-RF data synthesis. Our extensive real-world experiments highlight RFGen's effectiveness in diverse mmWave sensing applications, enhancing their generalization to unseen test cases without laborious data collection.
- Karan Ahuja, Yue Jiang, Mayank Goel, and Chris Harrison. 2021. Vid2Doppler: Synthesizing Doppler radar data from videos for training privacy-preserving activity recognition. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--10.Google Scholar
Digital Library
- Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. 36, 4 (2017), 97--1.Google ScholarDigital Library
- Pietro Barbiero, Giovanni Squillero, and Alberto Tonda. 2020. Modeling generalization in machine learning: A methodological and computational study. arXiv preprint arXiv:2006.15680 (2020).Google Scholar
- Dennis Barrett. 2017. Smarter robotics through mmwave radar sensing | electronic design. https://www.electronicdesign.com/markets/automotive/article/21805921/smarter-robotics-through-mmwave-radar-sensingGoogle Scholar
- Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. 2023. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22563--22575.Google ScholarCross Ref
- Hanqun Cao, Cheng Tan, Zhangyang Gao, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li. 2022. A survey on generative diffusion model. arXiv preprint arXiv:2209.02646 (2022).Google Scholar
- Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman. 2022. Soundspaces 2.0: A simulation platform for visual-acoustic learning. Advances in Neural Information Processing Systems (2022).Google Scholar
- Xingyu Chen, Zhengxiong Li, Biacheng Chen, Yi Zhu, Chris Xiaoxuan Lu, Zhengyu Peng, Feng Lin, Wenyao Xu, Kui Ren, and Chunming Qiao. 2022. MetaWave: Attacking mmWave Sensing with Meta-material-enhanced Tags. In The 30th Network and Distributed System Security (NDSS) Symposium 2023. The Internet Society.Google Scholar
- William H Clark IV, Steven Hauser, William C Headley, and Alan J Michaels. 2021. Training data augmentation for deep learning radio frequency systems. The Journal of Defense Modeling and Simulation 18, 3 (2021), 217--237.Google ScholarCross Ref
- Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. 2023. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).Google ScholarDigital Library
- Kaikai Deng, Dong Zhao, Qiaoyue Han, Zihan Zhang, Shuyue Wang, Anfu Zhou, and Huadong Ma. 2023. Midas: Generating mmWave Radar Data from Videos for Training Pervasive and Privacy-preserving Human Sensing Tasks. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 1 (2023), 1--26.Google ScholarDigital Library
- Yudi Dong and Yu-Dong Yao. 2020. Secure mmWave-radar-based speaker verification for IoT smart home. IEEE Internet of Things Journal 8, 5 (2020), 3500--3511.Google ScholarCross Ref
- Christer Ericson. 2004. Real-time collision detection. Crc Press.Google Scholar
- Gongfan Fang, Xinyin Ma, and Xinchao Wang. 2023. Structural Pruning for Diffusion Models. arXiv preprint arXiv:2305.10924 (2023).Google Scholar
- Nahuel E Garcia-D'Urso, Jorge Azorin-Lopez, and Andres Fuster-Guillo. 2023. Accurate Estimation of Parametric Models of the Human Body from 3D Point Clouds. In International Conference on Soft Computing Models in Industrial and Environmental Applications. Springer, 236--245.Google ScholarCross Ref
- Jian Gong, Xinyu Zhang, Kaixin Lin, Ju Ren, Yaoxue Zhang, and Wenxun Qiu. 2021. RF vital sign sensing under free body movement. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 3 (2021).Google ScholarDigital Library
- David J Griffiths. 2005. Introduction to electrodynamics.Google Scholar
- Yan Gu, Yong He, Kayvon Fatahalian, and Guy Blelloch. 2013. Efficient BVH construction via approximate agglomerative clustering. In Proceedings of the 5th High-Performance Graphics Conference. 81--88.Google ScholarDigital Library
- Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge, Ying Shan, and Mike Zheng Shou. 2023. Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models. arXiv:2305.18292Google Scholar
- Robert Alfred Herman. 1900. A treatise on geometrical optics. University Press.Google Scholar
- Rodrigo Hernangómez, Tristan Visentin, Lorenzo Servadei, Hamid Khodabakhshandeh, and Sławomir Stańczak. 2022. Improving Radar Human Activity Classification Using Synthetic Data with Image Transformation. Sensors 22, 4 (2022), 1519.Google ScholarCross Ref
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685Google Scholar
- Sajjad Hussain. 2017. Efficient ray-tracing algorithms for radio wave propagation in urban environments. Ph.D. Dissertation. Dublin City University.Google Scholar
- Cesar Iovescu and Sandeep Rao. 2023. The fundamentals of millimeter wave radar sensor.Google Scholar
- Alireza Javaheri, Catarina Brites, Fernando Pereira, and João Ascenso. 2020. A generalized Hausdorff distance based quality metric for point cloud geometry. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 1--6.Google ScholarCross Ref
- Belal Korany, Chitra R. Karanam, Hong Cai, and Yasamin Mostofi. 2019. XModal-ID: Using WiFi for Through-Wall Person Identification from Candidate Video Footage. In ACM Annual International Conference on Mobile Computing and Networking (MobiCom).Google ScholarDigital Library
- Hyeokhyen Kwon, Catherine Tong, Harish Haresamudram, Yan Gao, Gregory D Abowd, Nicholas D Lane, and Thomas Ploetz. 2020. IMUTube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020).Google ScholarDigital Library
- Xiangjun Li and Jianfei Cai. 2007. Robust transmission of JPEG2000 encoded images over packet loss channels. In 2007 IEEE International Conference on Multimedia and Expo. IEEE, 947--950.Google ScholarCross Ref
- Yadong Li, Dongheng Zhang, Jinbo Chen, Jinwei Wan, Dong Zhang, Yang Hu, Qibin Sun, and Yan Chen. 2022. Towards domain-independent and real-time gesture recognition using mmwave signal. IEEE Transactions on Mobile Computing (2022).Google ScholarDigital Library
- Zhengxiong Li, Baicheng Chen, Xingyu Chen, Huining Li, Chenhan Xu, Feng Lin, Chris Xiaoxuan Lu, Kui Ren, and Wenyao Xu. 2022. SpiralSpy: Exploring a stealthy and practical covert channel to attack air-gapped computing devices via mmWave sensing. In Proc. NDSS. 1--16.Google ScholarCross Ref
- Hao Ling, R-C Chou, and S-W Lee. 1989. Shooting and bouncing rays: Calculating the RCS of an arbitrarily shaped cavity. IEEE Transactions on Antennas and propagation 37, 2 (1989), 194--205.Google ScholarCross Ref
- Keith Man and Javaan Chahl. 2022. A Review of Synthetic Image Data and Its Use in Computer Vision. Journal of Imaging 8, 11 (2022), 310.Google ScholarCross Ref
- Sivabalan Manivasagam, Shenlong Wang, Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang, Wei-Chiu Ma, and Raquel Urtasun. 2020. Lidarsim: Realistic lidar simulation by leveraging the real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
- Marzuki Marzuki, Dea Kurnia Harysandi, Rini Oktaviani, Lisna Meylani, Mutya Vonnisa, Harmadi Harmadi, Hiroyuki Hashiguchi, Toyoshi Shimomai, L Luini, Sugeng Nugroho, et al. 2020. International Telecommunication Union-Radiocommunication Sector P. 837-6 and P. 837-7 performance to estimate Indonesian rainfall. TELKOMNIKA (Telecommunication Computing Electronics and Control) 18, 5 (2020), 2292--2303.Google ScholarCross Ref
- Morgan McGuire. 2017. Computer Graphics Archive. https://casual-effects.com/dataGoogle Scholar
- Facundo Mémoli and Guillermo Sapiro. 2004. Comparing point clouds. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing. 32--40.Google ScholarDigital Library
- Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. arXiv:2003.08934Google Scholar
- Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. 2023. Diffrf: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4328--4338.Google ScholarCross Ref
- Yong Niu, Yong Li, Depeng Jin, Li Su, and Athanasios V Vasilakos. 2015. A survey of millimeter wave communications (mmWave) for 5G: opportunities and challenges. Wireless networks 21 (2015), 2657--2676.Google Scholar
- OpenAI. 2023. GPT-4 technical report. arXiv (2023), 2303--08774.Google Scholar
- Amani Yousef Owda, Neil Salmon, Stuart William Harmer, Sergiy Shylo, Nicholas John Bowring, Nacer Ddine Rezgui, and Mamta Shah. 2017. Millimeter-wave emissivity as a metric for the non-contact diagnosis of human skin conditions. Bioelectromagnetics 38, 7 (2017), 559--569.Google ScholarCross Ref
- paperswithcode. 2023. 3D Human Pose Estimation Datasets. https://paperswithcode.com/task/3d-human-pose-estimationGoogle Scholar
- Kun Qian, Zhaoyuan He, and Xinyu Zhang. 2020. 3D point cloud generation with millimeter-wave radar. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (2020).Google ScholarDigital Library
- Kun Qian, Shilin Zhu, Xinyu Zhang, and Li Erran Li. 2021. Robust multimodal vehicle detection in foggy weather using complementary lidar and radar signals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
- Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10684--10695.Google ScholarCross Ref
- Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22500--22510.Google ScholarCross Ref
- Christian Schöffmann, Barnaba Ubezio, Christoph Böhm, Stephan Mühlbacher-Karrer, and Hubert Zangl. 2021. Virtual radar: Real-time millimeter-wave radar sensor simulation for perception-driven robotics. IEEE Robotics and Automation Letters 6, 3 (2021), 4704--4711.Google ScholarCross Ref
- Aristeidis Seretis and Costas D Sarris. 2021. An overview of machine learning techniques for radiowave propagation modeling. IEEE Transactions on Antennas and Propagation 70, 6 (2021), 3970--3985.Google ScholarCross Ref
- Peter Shirley and R Keith Morley. 2008. Realistic ray tracing. AK Peters, Ltd.Google Scholar
- William Sloane, Camillo Gentile, Mansoor Shafi, Jelena Senic, Philippa A. Martin, and Graeme K. Woodward. 2023. Measurement-Based Analysis of Millimeter-Wave Channel Sparsity. IEEE Antennas and Wireless Propagation Letters 22, 4 (2023).Google ScholarCross Ref
- Svengali75. 2023. Svengali75/ProtogenX53Photorealism. https://huggingface.co/Svengali75/ProtogenX53Photorealism Accessed: 2023-06-28.Google Scholar
- Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. 2022. Human motion diffusion model. arXiv preprint arXiv:2209.14916 (2022).Google Scholar
- Shelly Vishwakarma, Wenda Li, Chong Tang, Karl Woodbridge, Raviraj Adve, and Kevin Chetty. 2021. Neural style transfer enhanced training support for human activity recognition. arXiv preprint arXiv:2107.12821 (2021).Google Scholar
- Peng Wang and Manuel Vindiola. 2019. Data augmentation for blind signal classification. In MILCOM 2019-2019 IEEE Military Communications Conference (MILCOM). IEEE, 305--310.Google ScholarDigital Library
- Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612.Google ScholarDigital Library
- Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. 2022. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. bioRxiv (2022).Google Scholar
- Teng Wei and Xinyu Zhang. 2015. mtrack: High-precision passive tracking using millimeter wave radios. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking.Google ScholarDigital Library
- Teng Wei, Anfu Zhou, and Xinyu Zhang. 2017. Facilitating Robust 60 {GHz} Network Deployment By Sensing Ambient Reflectors. In USENIX Symposium on Networked Systems Design and Implementation (NSDI).Google Scholar
- Zhiqing Wei, Fengkai Zhang, Shuo Chang, Yangyang Liu, Huici Wu, and Zhiyong Feng. 2022. Mmwave radar and vision fusion for object detection in autonomous driving: A review. Sensors 22, 7 (2022), 2542.Google ScholarCross Ref
- Xuhai Xu, Xin Liu, Han Zhang, Weichen Wang, Subigya Nepal, Yasaman Sefidgar, Woosuk Seo, Kevin S Kuehn, Jeremy F Huckins, Margaret E Morris, et al. 2023. GLOBEM: Cross-Dataset Generalization of Longitudinal Human Behavior Modeling. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 4 (2023).Google Scholar
- Hongfei Xue, Yan Ju, Chenglin Miao, Yijiang Wang, Shiyang Wang, Aidong Zhang, and Lu Su. 2021. mmMesh: towards 3D real-time dynamic human mesh construction using millimeter-wave. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services. 269--282.Google ScholarDigital Library
- Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, and Tarek Abdelzaher. 2017. Deepsense: A unified deep learning framework for time-series mobile sensing data processing. In Proceedings of the 26th international conference on world wide web. 351--360.Google ScholarDigital Library
- Zhengqing Yun and Magdy F Iskander. 2015. Ray tracing for radio propagation modeling: Principles and applications. IEEE access 3 (2015), 1089--1100.Google Scholar
- Chenshuang Zhang, Chaoning Zhang, Sheng Zheng, Mengchun Zhang, Maryam Qamar, Sung-Ho Bae, and In So Kweon. 2023. A survey on audio diffusion models: Text to speech synthesis and enhancement in generative ai. arXiv preprint arXiv:2303.13336 2 (2023).Google Scholar
- Jie Zhang, Zhanyong Tang, Meng Li, Dingyi Fang, Petteri Nurmi, and Zheng Wang. 2018. CrossSense: Towards Cross-Site and Large-Scale WiFi Sensing. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking (MobiCom).Google ScholarDigital Library
- Lvmin Zhang and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 (2023).Google Scholar
- Longyu Zhang, Bote Han, Haiwei Dong, and Abdulmotaleb El Saddik. 2017. Development of an automatic 3D human head scanning-printing system. Multimedia Tools and Applications 76 (2017), 4381--4403.Google ScholarDigital Library
- Xiaotong Zhang, Zhenjiang Li, and Jin Zhang. 2022. Synthesized Millimeter-Waves for Human Motion Sensing. In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems. 377--390.Google ScholarDigital Library
- Anfu Zhou, Shaoqing Xu, Song Wang, Jingqi Huang, Shaoyuan Yang, Teng Wei, Xinyu Zhang, and Huadong Ma. 2019. Robot navigation in radio beam space: Leveraging robotic intelligence for seamless mmwave network coverage. In Proceedings of the ACM International Symposium on Mobile Ad Hoc Networking and Computing.Google ScholarDigital Library
- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision. 2223--2232.Google ScholarCross Ref
Index Terms
- RF Genesis: Zero-Shot Generalization of mmWave Sensing through Simulation-Based Data Synthesis and Generative Diffusion Models
Recommendations
Poster: Generalizing mmWave Sensing with Simulation Synthesis and Generative Models
HOTMOBILE '24: Proceedings of the 25th International Workshop on Mobile Computing Systems and ApplicationsWireless sensing through emerging millimeter-wave (mmWave) technology has gained traction recently, integrating extensively into human life and transitioning from research prototypes to standardization and commercialization, due to the high angular and ...
New CRLH-Based Planar Slotted Antennas with Helical Inductors for Wireless Communication Systems, RF-Circuits and Microwave Devices at UHF---SHF Bands
Two novel planar slotted-antennas (PSAs) are presented that exhibit good radiation characteristics at the UHF---SHF bands. The proposed antennas are constructed using metamaterial unit-cells constituted from capacitive slots etched in the radiating ...
Sensing the Physical World with RF: Self-Interferometry & Passive-Interferometry
S3'19: Proceedings of the 2019 on Wireless of the Students, by the Students, and for the Students WorkshopRF can provide a non-contact and non-line-of-sight of sensing of the physical world, therefore, it makes RF unique sensing modality that has found applications in automotive sensing, smart-home sensing, health monitoring, and many other applications. ...
Comments