Multi-modal pose estimation in XR applications leveraging integrated sensing and communication
Authors: 
Nabeel Nisar Bhat, Javad Sameri, Jakob Struye, Maria Torres Vega, Rafael Berkvens, Jeroen FamaeyAuthors Info & Claims
Nabeel Nisar Bhat, Javad Sameri, Jakob Struye, Maria Torres Vega, Rafael Berkvens, Jeroen FamaeyAuthors Info & ClaimsPages 261 - 267
Abstract
Mobile extended reality (XR) applications are anticipated to generate substantial traffic for 6G. Such applications not only require high data rate and low-latency transmissions, but also accurate and real-time pose estimation to enable interactive and immersive experiences. While sub-6 GHz signals have been exploited for pose estimation, they cannot cope up with multi-gigabit data rates required by XR applications. Instead, mobile communications at mmWave frequencies can potentially support data rates up to several giga-bits per second (Gbps) and, therefore, can be used to deliver XR content wirelessly to the Head-Mounted Display (HMD). Moreover, mmWave frequencies can offer improved sensing due to the large available bandwidth. Therefore, mmWave communications can play a crucial role in enabling device-free interactivity by offering both high-speed communication and accurate sensing capabilities. However, mmWave propagation characteristics are different from sub-6 GHz. Path loss plays a significant role, and can lead to degraded sensing performance. Therefore, our proposal supplements wireless sensing at mmWave frequencies with wireless electromyography (EMG) armbands. By capturing patterns of muscle activities, we can counteract the limitations of mmWave-based pose estimation, thereby enriching the granularity and precision of pose estimation. This paper proposes a conceptual architecture to achieve multi-modal pose estimation for XR applications. Early results highlight the shortcomings of mmWave-based sensing, and we identify future steps and opportunities on integration of both approaches.
References
[1]
Tianyang Bai and Robert W Heath. 2014. Analysis of self-body blocking effects in millimeter wave cellular networks. In 2014 48th Asilomar conference on signals, systems and computers. IEEE, 1921--1925.
[2]
Nabeel Nisar Bhat, Rafael Berkvens, and Jeroen Famaey. 2023. Gesture Recognition with mmWave Wi-Fi Access Points: Lessons Learned. In 2023 IEEE 24th International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 127--136.
[3]
Alejandro Blanco, Pablo Jiménez Mateo, Francesco Gringoli, and Joerg Widmer. 2022. Augmenting mmWave localization accuracy through sub-6 GHz on off-the-shelf devices. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services. 477--490.
[4]
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7291--7299.
[5]
Yuanhao Cui, Fan Liu, Xiaojun Jing, and Junsheng Mu. 2021. Integrating sensing and communications for ubiquitous IoT: Applications, trends, and challenges. IEEE Network 35, 5 (2021), 158--167.
[6]
Carlos De Lima, Didier Belot, Rafael Berkvens, Andre Bourdoux, Davide Dardari, Maxime Guillaud, Minna Isomursu, Elena-Simona Lohan, Yang Miao, Andre Noll Barreto, et al. 2021. Convergent communication, sensing and localization in 6G systems: An overview of technologies, opportunities and challenges. IEEE Access 9 (2021), 26902--26925.
[7]
Youjia Huang, Xingchen Yang, Yuefeng Li, Dalin Zhou, Keshi He, and Honghai Liu. 2017. Ultrasound-based sensing models for finger motion classification. IEEE journal of biomedical and health informatics 22, 5 (2017), 1395--1405.
[8]
Shuo Jiang, Qinghua Gao, Huaiyang Liu, and Peter B Shull. 2020. A novel, co-located EMG-FMG-sensing wearable armband for hand gesture recognition. Sensors and Actuators A: Physical 301 (2020), 111738.
[9]
Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L Iuzzolino, and Kazuhito Koishida. 2020. MMTM: Multimodal transfer module for CNN fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13289--13299.
[10]
Peiqi Kang, Jinxuan Li, Shuo Jiang, and Peter B Shull. 2022. Reduce System Redundancy and Optimize Sensor Disposition for EMG-IMU Multi-modal Fusion Human-Machine Interfaces with XAI. IEEE Transactions on Instrumentation and Measurement (2022).
[11]
Naveen Kumar Karnam, Shiv Ram Dubey, Anish Chand Turlapaty, and Balakrishna Gokaraju. 2022. EMGHandNet: A hybrid CNN and Bi-LSTM architecture for hand activity classification using surface EMG signals. Biocybernetics and Biomedical Engineering 42, 1 (2022), 325--340.
[12]
EunSu Kim, JaeWook Shin, YongSung Kwon, and BumYong Park. 2023. EMG-Based Dynamic Hand Gesture Recognition Using Edge AI for Human-Robot Interaction. Electronics 12, 7 (2023), 1541.
[13]
Toshiaki Koike-Akino, Pu Wang, Milutin Pajovic, Haijian Sun, and Philip V Orlik. 2020. Fingerprinting-based indoor localization with commercial MMWave Wi-Fi: A deep learning approach. IEEE Access 8 (2020), 84879--84892.
[14]
Filip Lemic, James Martin, Christopher Yarp, Douglas Chan, Vlado Handziski, Robert Brodersen, Gerhard Fettweis, Adam Wolisz, and John Wawrzynek. 2016. Localization as a feature of mmWave communication. In 2016 International Wireless Communications and Mobile Computing Conference (IWCMC). IEEE, 1033--1038.
[15]
Iliana Loi, Angeliki Grammatikaki, Panagiotis Tsinganos, Efe Bozkir, Dimitris Ampeliotis, Konstantinos Moustakas, Enkelejda Kasneci, and Athanassios Skodras. 2022. Proportional Myoelectric Control in a Virtual Reality Environment. In 2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP). IEEE, 1--5.
[16]
Yongsen Ma, Gang Zhou, Shuangquan Wang, Hongyang Zhao, and Woosub Jung. 2018. Signfi: Sign language recognition using Wi-Fi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1--21.
[17]
Francesca Meneghello, Domenico Garlisi, Nicolo Dal Fabbro, Ilenia Tinnirello, and Michele Rossi. 2022. SHARP: Environment and Person Independent Activity Recognition with Commodity IEEE 802.11 Access Points. IEEE Transactions on Mobile Computing (2022).
[18]
Sameera Palipana, David Rojas, Piyush Agrawal, and Dirk Pesch. 2018. FallDeFi: Ubiquitous fall detection using commodity Wi-Fi devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1--25.
[19]
Yili Ren, Zi Wang, Yichao Wang, Sheng Tan, Yingying Chen, and Jie Yang. 2022. Gopose: 3D human pose estimation using Wi-Fi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (2022), 1--25.
[20]
Francesco Restuccia. 2021. IEEE 802.11 bf: Toward ubiquitous Wi-Fi sensing. arXiv preprint arXiv:2103.14918 (2021).
[21]
Jakob Struye, Filip Lemic, and Jeroen Famaey. 2020. Towards ultra-low-latency mmwave Wi-Fi for multi-user interactive virtual reality. In GLOBECOM 2020-2020 IEEE Global Communications Conference. IEEE, 1--6.
[22]
Afroza Sultana, Farruk Ahmed, and Md Shafiul Alam. 2022. A systematic review on surface electromyography-based classification system for identifying hand and finger movements. Healthcare Analytics (2022), 100126.
[23]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1--9.
[24]
Turker Tuncer, Sengul Dogan, and Abdulhamit Subasi. 2022. Novel finger movement classification method based on multi-centered binary pattern using surface electromyogram signals. Biomedical Signal Processing and Control 71 (2022), 103153.
[25]
Bram van Berlo, Amany Elkelany, Tanir Ozcelebi, and Nirvana Meratnia. 2021. Millimeter wave sensing: A review of application pipelines and building blocks. IEEE Sensors Journal 21, 9 (2021), 10332--10368.
[26]
Juan Pablo Vásconez, Lorena Isabel Barona López, Ángel Leonardo Valdivieso Caraguay, and Marco E Benalcázar. 2022. Hand Gesture Recognition Using EMG-IMU Signals and Deep Q-Networks. Sensors 22, 24 (2022), 9613.
[27]
Jianyuan Yu, Pu Wang, Toshiaki Koike-Akino, Ye Wang, Philip V Orlik, and Haijian Sun. 2020. Human pose and seat occupancy classification with commercial MMWave Wi-Fi. In 2020 IEEE Globecom Workshops. IEEE, 1--6.
[28]
Tao Zhang, Tingyu Song, Daolin Chen, Tian Zhang, and Jie Zhuang. 2019. WiGrus: A Wi-Fi-based gesture recognition system using software-defined radio. IEEE Access 7 (2019), 131102--131113.
[29]
Zhengyou Zhang. 2012. Microsoft kinect sensor and its effect. IEEE multimedia 19, 2 (2012), 4--10.
[30]
Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. 2018. Through-wall human pose estimation using radio signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7356--7365.
[31]
Yue Zheng, Yi Zhang, Kun Qian, Guidong Zhang, Yunhao Liu, Chenshu Wu, and Zheng Yang. 2019. Zero-effort cross-domain gesture recognition with Wi-Fi. In Proceedings of the 17th annual international conference on mobile systems, applications, and services. 313--325.
[32]
Yunjiao Zhou, He Huang, Shenghai Yuan, Han Zou, Lihua Xie, and Jianfei Yang. 2023. MetaFi++: Wi-Fi-Enabled Transformer-based Human Pose Estimation for Metaverse Avatar Simulation. IEEE Internet of Things Journal (2023).
Index Terms
- Multi-modal pose estimation in XR applications leveraging integrated sensing and communication
Recommendations
Design, Simulation and Analysis of a High Gain Small Size Array Antenna for 5G Wireless Communication
AbstractMicrostrip antennas have become a part and parcel of today’s wireless communication world because of their low profile, low cost and ease of fabrication in the circuit boards. But poor performances like narrow bandwidth, low power handling ...
Side-lobe Can Know More: Towards Simultaneous Communication and Sensing for mmWave
Thanks to the wide bandwidth, large antenna array, and short wavelength, millimeter wave (mmWave) has superior performance in both communication and sensing. Thus, the integration of sensing and communication is a developing trend for the mmWave band. ...
Performance Analysis of an Extended Sierpinski Gasket Fractal Antenna for Millimeter-wave Femtocells Applications
AbstractA performance study on the design and analysis of an extended Sierpinski gasket fractal antenna for mm-wave femtocell applications were implemented. The initial analysis includes the design of different stages of basic Sierpinski gasket fractal ...
Comments
Information & Contributors
Information
Published In
October 2023
83 pages
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 06 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- FWO
Conference
ImmerCom '23
Sponsor:
ImmerCom '23: 1st ACM Workshop on Mobile Immersive Computing, Networking, and Systems
October 6, 2023
Madrid, Spain
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 271Total Downloads
- Downloads (Last 12 months)153
- Downloads (Last 6 weeks)14
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in