skip to main content
10.1145/3636534.3696211acmconferencesArticle/Chapter ViewAbstractPublication PagesmobicomConference Proceedingsconference-collections
research-article

Towards In-context Environment Sensing for Mobile Augmented Reality

Published: 04 December 2024 Publication History

Abstract

Environment sensing is a fundamental task in mobile augmented reality (AR). However, on-device sensing and computing resources often limit mobile AR sensing capability, making high-quality environment sensing challenging to achieve. In recent years, in-context sensing, a new sensing system design paradigm, has emerged with the promise of achieving accurate, efficient, and robust sensing results. In this work, we first formally define the in-context sensing design paradigm. We summarize its primary challenges as the uncertainty of environmental information availability. To quantify the impact of sensing context data, we present two in-depth case studies that show how it can impact different aspects of mobile AR sensing systems.

References

[1]
G. Baruch, Z. Chen, A. Dehghan, T. Dimry, Y. Feigin, P. Fu, T. Gebauer, B. Joffe, D. Kurz, A. Schwartz, and E. Shulman. ARKitscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[2]
S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller. Zoedepth: Zeroshot transfer by combining relative and metric depth. arXiv:2302.12288.
[3]
C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874--1890, 2021.
[4]
J. Carmigniani, B. Furht, M. Anisetti, P. Ceravolo, E. Damiani, and M. Ivkovic. Augmented reality technologies, systems and applications. Multimedia tools and applications, 51:341--377, 2011.
[5]
A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
[6]
C. Chen, J. Wei, C. Peng, and H. Qin. Depth-quality-aware salient object detection. IEEE Transactions on Image Processing, 30:2350--2363, 2021.
[7]
L. Duan, Y. Chen, and M. Gorlatova. Demo abstract: Biguide: A bilevel data acquisition guidance for object detection on mobile devices. In Proceedings of the 22nd International Conference on Information Processing in Sensor Networks, pages 368--369, 2023.
[8]
A. Ganj, H. Su, and T. Guo. Hybriddepth: Robust depth fusion for mobile ar by leveraging depth from focus and single-image priors, 2024.
[9]
A. Ganj, Y. Zhao, F. Galbiati, and T. Guo. Toward scalable and controllable ar experimentation. In Proceedings of the 1st ACM Workshop on Mobile Immersive Computing, Networking, and Systems, ImmerCom '23, page 237--246, New York, NY, USA, 2023. Association for Computing Machinery.
[10]
A. Ganj, Y. Zhao, H. Su, and T. Guo. Mobile ar depth estimation: Challenges & prospects. In Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications, HOTMOBILE '24, page 21--26, New York, NY, USA, 2024. Association for Computing Machinery.
[11]
V. Guizilini, I. Vasiljevic, D. Chen, R. Ambrus, and A. Gaidon. Towards zero-shot scale-aware monocular depth estimation. In ICCV, 2023.
[12]
M. Hu, W. Yin, C. Zhang, Z. Cai, X. Long, H. Chen, K. Wang, G. Yu, C. Shen, and S. Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint arXiv:2404.15506, 2024.
[13]
M. Hu, W. Yin, C. Zhang, Z. Cai, X. Long, H. Chen, K. Wang, G. Yu, C. Shen, and S. Shen. A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. 2024.
[14]
T. Jin, S. Wu, M. Dasari, K. Apicharttrisorn, and A. Rowe. Stagear: Markerless mobile phone localization for ar in live events. In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pages 1000--1010. IEEE, 2024.
[15]
C. LeGendre, W.-C. Ma, R. Pandey, S. Fanello, C. Rhemann, J. Dourgarian, J. Busch, and P. Debevec. Learning illumination from diverse portraits. In SIGGRAPH Asia 2020 Technical Communications, SA '20, New York, NY, USA, 2020. Association for Computing Machinery.
[16]
J. Li, H. Li, and Y. Matsushita. Lighting, reflectance and geometry estimation from 360 panoramic stereo. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10586--10595. IEEE, 2021.
[17]
A. Rosinol, M. Abate, Y. Chang, and L. Carlone. Kimera: an open-source library for real-time metric-semantic localization and mapping. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2020.
[18]
K. Sartipi, T. Do, T. Ke, K. Vuong, and S. I. Roumeliotis. Deep depth estimation from visual-inertial slam. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
[19]
T. Scargill, S. Eom, Y. Chen, and M. Gorlatova. Ambient intelligence for next-generation ar. arXiv preprint arXiv:2303.12968, 2023.
[20]
T. Starner, B. Schiele, and A. Pentland. Visual contextual awareness in wearable computing. In Digest of Papers. Second International Symposium on Wearable Computers (Cat. No. 98EX215).
[21]
W. Van Gansbeke, D. Neven, B. De Brabandere, and L. Van Gool. Sparse and noisy lidar completion with rgb guidance and uncertainty. In 2019 16th International Conference on Machine Vision Applications (MVA).
[22]
B. Wen and K. E. Bekris. Bundletrack: 6d pose tracking for novel objects without instance or category-level 3d models. In IEEE/RSJ International Conference on Intelligent Robots and Systems.
[23]
B. Wen, J. Tremblay, V. Blukis, S. Tyree, T. Muller, A. Evans, D. Fox, J. Kautz, and S. Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. CVPR, 2023.
[24]
L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024.
[25]
R. Ye, W. Xu, H. Fu, R. K. Jenamani, V. Nguyen, C. Lu, K. Dimitropoulou, and T. Bhattacharjee. Rcareworld: A human-centric simulation world for caregiving robots. IROS, 2022.
[26]
W. Yin, C. Zhang, H. Chen, Z. Cai, G. Yu, K. Wang, X. Chen, and C. Shen. Metric3d: Towards zero-shot metric 3d prediction from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9043--9053, 2023.
[27]
Y. Zhang, T. Scargill, A. Vaishnav, G. Premsankar, M. Di Francesco, and M. Gorlatova. Indepth: Real-time depth inpainting for mobile augmented reality. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 6(1), mar 2022.
[28]
Z. Zhang, S. Qiao, C. Xie, W. Shen, B. Wang, and A. L. Yuille. Single-shot object detection with enriched semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5813--5821, 2018.
[29]
Y. Zhao and T. Guo. Pointar: Efficient lighting estimation for mobile augmented reality. In European Conference on Computer Vision, pages 678--693. Springer, 2020.
[30]
Y. Zhao and T. Guo. Xihe: A 3d vision-based lighting estimation framework for mobile augmented reality. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '21, pages 28--40, 2021.
[31]
Y. Zhao, C. Ma, H. Huang, and T. Guo. Litar: Visually coherent lighting for mobile augmented reality. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6(3):1--29, 2022.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ACM MobiCom '24: Proceedings of the 30th Annual International Conference on Mobile Computing and Networking
December 2024
2476 pages
ISBN:9798400704895
DOI:10.1145/3636534
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2024

Check for updates

Author Tags

  1. mobile AR
  2. context awareness
  3. environment understanding

Qualifiers

  • Research-article

Funding Sources

Conference

ACM MobiCom '24
Sponsor:

Acceptance Rates

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 24
    Total Downloads
  • Downloads (Last 12 months)24
  • Downloads (Last 6 weeks)8
Reflects downloads up to 08 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media