skip to main content
10.1145/3597065.3597449acmconferencesArticle/Chapter ViewAbstractPublication PagesmobicomConference Proceedingsconference-collections
research-article

Vision and Radar Multimodal Aided Beam Prediction: Facilitating Metaverse Development

Published: 20 June 2023 Publication History

Abstract

The metaverse requires enhanced communication rates and increased capacity, which can be attained by utilizing millimeter wave (mmWave) and terahertz (THz) communication systems along with large-scale antenna arrays. However, these systems come with a considerable beam training overhead. To address this challenge, this study proposes a novel multimodal deep learning framework based on 3D convolutional transformers for sensor-assisted beam prediction. Our approach utilizes both vision and radar data, resulting in quick and precise beam prediction. Our proposed scheme demonstrates more than 78% top-3 beam prediction accuracy in four different communication scenarios. Furthermore, the total prediction accuracy of our proposed framework is 85.6%, which is nearly 10% higher than using only single-sensory data. Our proposed solution effectively reduces beam training overhead and provides reliable communication support for high-mobility environments.

References

[1]
J. N. Njoku, C. Ifeanyi Nwakanma and D. -S. Kim, "The Role of 5G Wireless Communication System in the Metaverse," 2022 27th Asia Pacific Conference on Communications (APCC), Jeju Island, Korea, Republic of, 2022, pp. 290--294
[2]
H. Yuan, C. Wang, Y. Li, N. Liu and G. Cui, "The design of array antennas used for Massive MIMO system in the fifth generation mobile communication," 2016 11th International Symposium on Antennas, Propagation and EM Theory (ISAPE), Guilin, China, 2016, pp. 75--78
[3]
Y. Sun and C. Qi, "Analog Beamforming and Combining Based on Codebook in Millimeter Wave Massive MIMO Communications," GLOBECOM 2017-2017 IEEE Global Communications Conference, Singapore, 2017, pp. 1--6
[4]
H. -L. Chiang, W. Rave, T. Kadur and G. Fettweis, "A low-complexity beamforming method by orthogonal codebooks for millimeterwave links," 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA, 2017, pp. 3375--3379
[5]
Z. Xiao, T. He, P. Xia and X. -G. Xia, "Hierarchical Codebook Design for Beam-forming Training in Millimeter-Wave Communication," IEEE Transactions on Wireless Communications (TWC), vol. 15, no. 5, pp. 3380--3392, May 2016
[6]
S. Rezaie, J. Morais, E. de Carvalho, A. Alkhateeb and C. N. Manchón, "Location- and Orientation-aware Millimeter Wave Beam Selection for Multi -Panel Antenna Devices," GLOBECOM 2022 - 2022 IEEE Global Communications Conference, Rio de Janeiro, Brazil, 2022, pp. 597--602
[7]
M. Alrabeiah, A. Hredzak and A. Alkhateeb, "Millimeter Wave Base Stations with Cameras: Vision-Aided Beam and Blockage Prediction," 2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring), Antwerp, Belgium, 2020, pp. 1--5
[8]
F. Liu, W. Yuan, C. Masouros and J. Yuan, "Radar-Assisted Predictive Beamforming for Vehicular Links: Communication Served by Sensing," IEEE Transactions on Wireless Communications (TWC), vol. 19, no. 11, pp. 7704--7719, Nov. 2020
[9]
G. Charan, T. Osman, A. Hredzak, N. Thawdar and A. Alkhateeb, "Vision-Position Multi-Modal Beam Prediction Using Real Millimeter Wave Datasets," 2022 IEEE Wireless Communications and Networking Conference (WCNC), Austin, TX, USA, 2022, pp. 2727--2731
[10]
G. Charan, A. Hredzak, C. Stoddard, B. Berrey, M. Seth, H. Nunez, and A. Alkhateeb, "Towards real-world 6G drone communication: Position and camera aided beam prediction," GLOBECOM 2022. [Online]. Available: https://arxiv.org/abs/2205.12187
[11]
A. Alkhateeb, G. Charan, T. Osman, A. Hredzak, and N. Srinivas, "DeepSense 6G: Large-scale real-world multi-modal sensing and communication datasets," to be available on arXiv, 2022. [Online]. Available: https://www.DeepSense6G.net
[12]
U. Demirhan and A. Alkhateeb, "Radar aided 6g beam prediction: Deep learning algorithms and real-world demonstration," IEEE Wireless Communications and Networking Conference (WCNC), 2022, pp. 2655--2660.

Cited By

View all
  • (2024)CommRad: Context-Aware Sensing-Driven Millimeter-Wave NetworksProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699363(633-646)Online publication date: 4-Nov-2024
  • (2024)Sensing-Assisted High Reliable Communication: A Transformer-Based Beamforming ApproachIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2024.340585918:5(782-795)Online publication date: Jul-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISACom '23: Proceedings of the 2nd Workshop on Integrated Sensing and Communications for Metaverse
June 2023
41 pages
ISBN:9798400702150
DOI:10.1145/3597065
  • Co-chairs:
  • Yuanhao Cui,
  • Pengyuan Zhou,
  • Fan Liu,
  • Christos Masouros
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 June 2023

Check for updates

Author Tags

  1. beam prediction
  2. multimodal
  3. metaverse
  4. deep learning

Qualifiers

  • Research-article

Conference

ISACom '23
Sponsor:

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)76
  • Downloads (Last 6 weeks)8
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)CommRad: Context-Aware Sensing-Driven Millimeter-Wave NetworksProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699363(633-646)Online publication date: 4-Nov-2024
  • (2024)Sensing-Assisted High Reliable Communication: A Transformer-Based Beamforming ApproachIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2024.340585918:5(782-795)Online publication date: Jul-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media