skip to main content
10.1145/3560905.3568546acmconferencesArticle/Chapter ViewAbstractPublication PagessensysConference Proceedingsconference-collections
research-article
Open access

Breaking Edge Shackles: Infrastructure-Free Collaborative Mobile Augmented Reality

Published: 24 January 2023 Publication History

Abstract

Collaborative AR applications are gaining popularity, but have heavy computing requirements for identifying and tracking AR devices and objects in the ecosystem. Prior AR frameworks typically rely on edge infrastructure to offload AR's compute-heavy tasks. However, such infrastructure may not always be available, and continuously running AR computations on user devices can rapidly drain battery and impact application longevity. In this work, we enable infrastructure-free mobile AR with a low energy footprint, by using collaborative time slicing to distribute compute-heavy AR tasks across user devices. Realizing this idea is challenging because distributed execution can result in inconsistent synchronization of the AR virtual overlays. Our framework, FreeAR, tackles this with novel lightweight techniques for tightly synchronized virtual overlay placements across user views, and low latency recovery upon disruptions. We prototype FreeAR on Android and show that it can improve the virtual overlay positioning accuracy (with respect to the IOU metric) by up to 78%, relative to state-of-the-art collaborative AR systems, while also reducing power by up to 60% relative to a direct application of those prior solutions.

References

[1]
Fawad Ahmad, Hang Qiu, Ray Eells, Fan Bai, and Ramesh Govindan. 2020. Carmap: Fast 3d feature map updates for automobiles. In 17th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 20). USENIX Association, Santa Clara, CA, 1063--1081.
[2]
Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. 2021. Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild With Pose Annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[3]
Android. [n. d.]. Wi-Fi Direct (peer-to-peer or P2P) overview. https://developer.android.com/guide/topics/connectivity/wifip2p.
[4]
Android. 2022. Android Sensor: Linear Acceleration. https://developer.android.com/reference/android/hardware/Sensor#TYPE_LINEAR_ACCELERATION. Accessed: 2022-09-30.
[5]
Kittipat Apicharttrisorn, Bharath Balasubramanian, Jiasi Chen, Rajarajan Sivaraj, Yi-Zhen Tsai, Rittwik Jana, Srikanth Krishnamurthy, Tuyen Tran, and Yu Zhou. 2020. Characterization of Multi-User Augmented Reality over Cellular Networks. In 2020 17th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). 1--9.
[6]
Kittipat Apicharttrisorn, Jiasi Chen, Vyas Sekar, Anthony Rowe, and Srikanth V. Krishnamurthy. 2022. FreeAR Website. https://sites.google.com/view/infra-free-ar/home.
[7]
Kittipat Apicharttrisorn, Xukan Ran, Jiasi Chen, Srikanth V. Krishnamurthy, and Amit K. Roy-Chowdhury. 2019. Frugal Following: Power Thrifty Object Detection and Tracking for Mobile Augmented Reality. In Conference on Embedded Networked Sensor Systems (New York, New York) (SenSys). ACM, New York, NY, USA.
[8]
Apple. [n. d.]. Creating a Multiuser AR Experience. https://developer.apple.com/documentation/arkit/creating_a_multiuser_ar_experience.
[9]
Ali J. Ben Ali, Zakieh Sadat Hashemifar, and Karthik Dantu. 2020. Edge-SLAM: Edge-Assisted Visual Simultaneous Localization and Mapping. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services (MobiSys '20). ACM, New York, NY, USA.
[10]
S. Benhimane and E. Malis. 2004. Real-time image-based tracking of planes using efficient second-order minimization. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
[11]
G. Bradski. 2000. The OpenCV Library. Dr. Dobb's Journal of Software Tools (2000).
[12]
L. Cehovin, A. Leonardis, and M. Kristan. 2016. Visual Object Tracking Performance Measures Revisited. IEEE Transactions on Image Processing 25, 3 (March 2016), 1261--1274.
[13]
Kaifei Chen, Tong Li, Hyung-Sin Kim, David E Culler, and Randy H Katz. 2018. MARVEL: Enabling Mobile Augmented Reality with Low Energy and Low Latency. In Conference on Embedded Networked Sensor Systems (SenSys). ACM, 292--304.
[14]
Tiffany Yu-Han Chen, Lenin Ravindranath, Shuo Deng, Paramvir Bahl, and Hari Balakrishnan. 2015. Glimpse: Continuous, real-time object recognition on mobile devices. ACM SenSys (2015).
[15]
Titus Cieslewski, Siddharth Choudhary, and Davide Scaramuzza. 2018. Data-Efficient Decentralized Visual SLAM. In 2018 IEEE International Conference on Robotics and Automation (ICRA). 2466--2473.
[16]
Alexander Cunningham, Vadim Indelman, and Frank Dellaert. 2013. DDF-SAM 2.0: Consistent distributed smoothing and mapping. In 2013 IEEE International Conference on Robotics and Automation. 5220--5227.
[17]
Alexander Cunningham, Manohar Paluri, and Frank Dellaert. 2010. DDF-SAM: Fully distributed SLAM using Constrained Factor Graphs. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems. 3025--3030.
[18]
Jon Dugan, Seth Elliott, Jeff Mah, Bruce A.and Poskanzer, and Kaustubh Prabhu. 2022. iPerf - The ultimate speed test tool for TCP, UDP and SCTP. https://iperf.fr/.
[19]
Dorian Galvez-López and Juan D. Tardos. 2012. Bags of Binary Words for Fast Place Recognition in Image Sequences. IEEE Transactions on Robotics (2012).
[20]
Google. [n. d.]. MediaPipe Objectron. https://google.github.io/mediapipe/solutions/objectron.html.
[21]
Google. 2018. Share AR Experiences with Cloud Anchors. https://developers.google.com/ar/develop/java/cloud-anchors/cloud-anchors-overview-android.
[22]
Google. 2022. Google Just a Line. https://justaline.withgoogle.com/. Accessed: 2022-09-30.
[23]
Yongjie Guan, Xueyu Hou, Nan Wu, Bo Han, and Tao Han. 2022. Realtime 3D Object Detection for Headsets. arXiv preprint arXiv:2201.08812 (2022).
[24]
Kiryong Ha, Zhuo Chen, Wenlu Hu, Wolfgang Richter, Padmanabhan Pillai, and Mahadev Satyanarayanan. 2014. Towards wearable cognitive assistance. ACM MobiSys (2014).
[25]
Seongwon Han, Sungwon Yang, Jihyoung Kim, and Mario Gerla. 2012. Eye-Guardian: A Framework of Eye Tracking and Blink Detection for Mobile Device Users. In Proceedings of the Twelfth Workshop on Mobile Computing Systems and Applications (San Diego, California) (HotMobile '12). Association for Computing Machinery, New York, NY, USA, Article 6, 6 pages.
[26]
Puneet Jain, Justin Manweiler, and Romit Roy Choudhury. 2016. Low Bandwidth Offload for Mobile AR. ACM CoNEXT (2016).
[27]
Kaleb. 2022. FSensor. https://github.com/KalebKE/FSensor. Accessed: 2022-09-30.
[28]
Marco Karrer, Patrik Schmuck, and Margarita Chli. 2018. CVI-SLAM---Collaborative Visual-Inertial SLAM. IEEE Robotics and Automation Letters 3, 4 (2018), 2762--2769.
[29]
Kris Kitani. [n. d.]. Camera Matrix. http://www.cs.cmu.edu/~16385/s17/Slides/11.1_Camera_matrix.pdf.
[30]
Laurent Kneip, Hongdong Li, and Yongduek Seo. 2014. UPnP: An Optimal O(n) Solution to the Absolute Pose Problem with Universal Applicability. In Computer Vision - ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.).
[31]
Steven LaValle. [n. d.]. Virtual Reality. Cambridge University Press.
[32]
Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. 2009. EPnP: An Accurate O(n) Solution to the PnP Problem. Int. J. Comput. Vision (2009).
[33]
Peiliang Li, Tong Qin, Botao Hu, Fengyuan Zhu, and Shaojie Shen. 2017. Monocular Visual-Inertial State Estimation for Mobile Augmented Reality. In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).
[34]
Pengpeng Liang, Yifan Wu, Hu Lu, Liming Wang, Chunyuan Liao, and Haibin Ling. 2018. Planar Object Tracking in the Wild: A Benchmark. In 2018 IEEE International Conference on Robotics and Automation (ICRA).
[35]
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. CoRR abs/1405.0312 (2014). arXiv:1405.0312 http://arxiv.org/abs/1405.0312
[36]
Luyang Liu and Marco Gruteser. 2021. EdgeSharing: Edge Assisted Real-time Localization and Object Sharing in Urban Streets. In IEEE INFOCOM 2021.
[37]
Luyang Liu, Hongyu Li, and Marco Gruteser. 2019. Edge Assisted Real-time Object Detection for Mobile Augmented Reality. ACM MobiCom (2019).
[38]
Zida Liu, Guohao Lan, Jovan Stojkovic, Yunfan Zhang, Carlee Joe-Wong, and Maria Gorlatova. 2020. CollabAR: Edge-assisted Collaborative Image Recognition for Mobile Augmented Reality. In 2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN).
[39]
Giuseppe Loianno, Yash Mulgaonkar, Chris Brunner, Dheeraj Ahuja, Arvind Ramanandan, Murali Chari, Serafin Diaz, and Vijay Kumar. 2016. A swarm of flying smartphones. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 1681--1688.
[40]
Microsoft. [n. d.]. HoloLens 2. https://www.microsoft.com/en-us/hololens/hardware.
[41]
Microsoft. 2018. Shared experiences in Unity. https://docs.microsoft.com/en-us/windows/mixed-reality/shared-experiences-in-unity.
[42]
Raúl Mur-Artal and Juan D. Tardós. 2017. ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras. IEEE Transactions on Robotics 33, 5 (2017), 1255--1262.
[43]
Jannis Möller. 2019. VINS-Mobile-Android. https://github.com/jannismoeller/VINS-Mobile-Android.
[44]
OpenCV. [n. d.]. Cascade Classifier. https://docs.opencv.org/3.4/db/d28/tutorial_cascade_classifier.html.
[45]
OpenCV. 2022. Basic concepts of the homography explained with code. https://docs.opencv.org/4.x/d9/dab/tutorial_homography.html. Accessed: 2022-09-30.
[46]
OpenCV. 2022. Object Detection with Template Matching. https://docs.opencv.org/3.4.16/df/dfb/group__imgproc__object.html. Accessed: 2022-09-30.
[47]
Alexander Pacha. [n. d.]. Sensor Fusion Demo. https://github.com/apacha/sensor-fusion-demo.
[48]
Hang Qiu, Fawad Ahmad, Fan Bai, Marco Gruteser, and Ramesh Govindan. 2018. AVR: Augmented vehicular reality. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. 81--95.
[49]
Xukan Ran, Haoliang Chen, Zhenming Liu, and Jiasi Chen. 2018. DeepDecision: A Mobile Deep Learning Framework for Edge Video Analytics. IEEE INFOCOM (2018).
[50]
Xukan Ran, Carter Slocum, Yi-Zhen Tsai, Kittipat Apicharttrisorn, Maria Gorlatova, and Jiasi Chen. 2020. Multi-user augmented reality with communication efficient and spatially consistent virtual objects. In ACM CoNEXT. 386--398.
[51]
Matthew Reynolds. [n. d.]. Pokemon Go Buddy Adventure explained - how to get hearts, excited Buddies, and all Buddy level rewards including Best Buddy explained. https://www.eurogamer.net/articles/2019-12-19-pokemon-go-buddy-adventure-play-excited-6002.
[52]
Patrik Schmuck and Margarita Chli. 2019. CCM-SLAM: Robust and efficient centralized collaborative monocular simultaneous localization and mapping for robotic teams. Journal of Field Robotics (2019).
[53]
Sheng Shen, Mahanth Gowda, and Romit Roy Choudhury. 2018. Closing the Gaps in Inertial Motion Tracking. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking (MobiCom '18). ACM, New York, NY, USA.
[54]
Arno Solin, Santiago Cortes, Esa Rahtu, and Juho Kannala. 2018. Inertial Odometry on Handheld Smartphones. In 2018 21st International Conference on Information Fusion (FUSION).
[55]
Spectacles. [n. d.]. New Spectables AR Glasses. https://www.spectacles.com/new-spectacles/.
[56]
Christian Szegedy, Alexander Toshev, and Dumitru Erhan. 2013. Deep neural networks for object detection. Advances in neural information processing systems 26 (2013).
[57]
Richard Szeliski. 2010. Computer vision: algorithms and applications. Springer Science & Business Media.
[58]
Mingxing Tan, Ruoming Pang, and Quoc V. Le. 2020. EfficientDet: Scalable and Efficient Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[59]
Zhenjun Tang, Y. Dai, and X. Zhang. 2012. Perceptual hashing for color images using invariant moments. Applied Mathematics and Information Sciences 6 (04 2012), 643S--650S.
[60]
TensorFlow. [n. d.]. TensorFlow Lite Object Detection Example. https://www.tensorflow.org/lite/examples/object_detection/overview.
[61]
TFHub.dev. [n. d.]. Image object detection. https://tfhub.dev/tensorflow/efficientdet/lite2/detection/1.
[62]
Ramona Trestian, Arghir-Nicolae Moldovan, Olga Ormond, and Gabriel-Miro Muntean. 2012. Energy consumption analysis of video streaming to Android mobile devices. In 2012 IEEE Network Operations and Management Symposium.
[63]
Maarten Van Steen and Andrew S Tanenbaum. 2017. Distributed systems. Maarten van Steen Leiden, The Netherlands.
[64]
Ana Villanueva, Zhengzhe Zhu, Ziyi Liu, Kylie Peppler, Thomas Redick, and Karthik Ramani. 2020. Meta-AR-App: An Authoring Platform for Collaborative Augmented Reality in STEM Classrooms. ACM.
[65]
VUZIX. [n. d.]. VUZIX BLADE UPGRADED SMART GLASSES. https://www.vuzix.com/products/vuzix-blade-smart-glasses-upgraded.
[66]
Yihong Wu and Zhanyi Hu. 2006. PnP problem revisited. Journal of Mathematical Imaging and Vision 24, 1 (2006), 131--141.
[67]
Wenxiao Zhang, Bo Han, and Pan Hui. 2018. Jaguar: Low Latency Mobile Augmented Reality with Flexible Tracking. In International Conference on Multimedia. ACM, 355--363.
[68]
Wenxiao Zhang, Bo Han, Pan Hui, Vijay Gopalakrishnan, Eric Zavesky, and Feng Qian. 2018. CARS: Collaborative Augmented Reality for Socialization. ACM HotMobile (2018).
[69]
Yinqiang Zheng, Yubin Kuang, Shigeki Sugimoto, Kalle Åström, and Masatoshi Okutomi. 2013. Revisiting the PnP Problem: A Fast, General and Optimal Solution. In 2013 IEEE International Conference on Computer Vision.
[70]
Danping Zou and Ping Tan. 2012. Coslam: Collaborative visual slam in dynamic environments. IEEE transactions on pattern analysis and machine intelligence 35, 2 (2012), 354--366.
[71]
Longhao Zou, Ali Javed, and Gabriel-Miro Muntean. 2017. Smart mobile device power consumption measurement for video streaming in wireless environments: WiFi vs. LTE. In 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB). 1--6.

Cited By

View all
  • (2024)Design of Secure and Efficient Authentication Protocol for Edge Computing-Based Augmented Reality EnvironmentsElectronics10.3390/electronics1303055113:3(551)Online publication date: 30-Jan-2024
  • (2024)Reducing End-to-End Latency of Trigger-Action IoT Programs on Containerized Edge PlatformsIEEE Transactions on Mobile Computing10.1109/TMC.2024.343953323:12(13979-13990)Online publication date: Dec-2024
  • (2024)HarvAR: Mobile Augmented-Reality-Assisted Photovoltaic Energy-Harvesting Sensor ManagementIEEE Internet of Things Journal10.1109/JIOT.2024.340216811:17(28591-28604)Online publication date: 1-Sep-2024
  • Show More Cited By

Index Terms

  1. Breaking Edge Shackles: Infrastructure-Free Collaborative Mobile Augmented Reality

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SenSys '22: Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems
    November 2022
    1280 pages
    ISBN:9781450398862
    DOI:10.1145/3560905
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 24 January 2023

    Check for updates

    Author Tags

    1. energy efficiency
    2. mobile augmented reality
    3. object detection and tracking
    4. simultaneous localization and mapping

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    Acceptance Rates

    SenSys '22 Paper Acceptance Rate 52 of 187 submissions, 28%;
    Overall Acceptance Rate 174 of 867 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)408
    • Downloads (Last 6 weeks)42
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Design of Secure and Efficient Authentication Protocol for Edge Computing-Based Augmented Reality EnvironmentsElectronics10.3390/electronics1303055113:3(551)Online publication date: 30-Jan-2024
    • (2024)Reducing End-to-End Latency of Trigger-Action IoT Programs on Containerized Edge PlatformsIEEE Transactions on Mobile Computing10.1109/TMC.2024.343953323:12(13979-13990)Online publication date: Dec-2024
    • (2024)HarvAR: Mobile Augmented-Reality-Assisted Photovoltaic Energy-Harvesting Sensor ManagementIEEE Internet of Things Journal10.1109/JIOT.2024.340216811:17(28591-28604)Online publication date: 1-Sep-2024
    • (2023)Toward Scalable and Controllable AR ExperimentationProceedings of the 1st ACM Workshop on Mobile Immersive Computing, Networking, and Systems10.1145/3615452.3617941(237-246)Online publication date: 6-Oct-2023
    • (2023)The Power of Asynchronous SLAM in Multi-User AR over Cellular Networks: A Measurement StudyProceedings of the 2023 Workshop on Emerging Multimedia Systems10.1145/3609395.3610598(34-40)Online publication date: 10-Sep-2023
    • (2023)Networked Architectures for Localization-Based Multi-User Augmented RealityIEEE Communications Magazine10.1109/MCOM.003.230027561:12(104-110)Online publication date: 28-Dec-2023

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media