skip to main content
10.1145/3477083.3480155acmconferencesArticle/Chapter ViewAbstractPublication PagesmobicomConference Proceedingsconference-collections
research-article

Characterizing real-time dense point cloud capture and streaming on mobile devices

Published: 25 October 2021 Publication History

Abstract

Point clouds are a dense compilation of millions of points that can advance content creation and interaction in various emerging applications such as Augmented Reality (AR). However, point clouds consist of per-point real-world spatial and color information that are too computationally intensive to meet real-time specifications, especially on mobile devices. To stream dense point cloud (PtCl) to mobile devices, existing solutions encode pre-captured point clouds, yet with PtCl capturing treated as a separate offline operation.
To discover more insights, we combine PtCl capturing and streaming as an entire pipeline and build a research prototype to study the bottlenecks of its real-time usage on mobile devices, consisting of a depth sensor with high precision and resolution, an edge-computing development board, and a smartphone. In a custom Unity app, we monitor the latency of each operation from the capturing to the rendering, as well as the energy efficiency of the board and the smartphone working at different point cloud resolutions. Results reveal that a toolset helping users efficiently capture, stream, and process color and depth data is the key enabler to real-time PtCl capturing and streaming on mobile devices.

References

[1]
Adobe. 2021. High-quality, network-efficient HTTP streaming. https://www.adobe.com/products/hds-dynamic-streaming.html.
[2]
I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 2016. 3D Semantic Parsing of Large-Scale Indoor Spaces. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1534--1543.
[3]
Cyrus S. Bamji, Swati Mehta, Barry Thompson, Tamer Elkhatib, Stefan Wurster, Onur Akkaya, Andrew Payne, John Godbaz, Mike Fenton, Vijay Rajasekaran, Larry Prather, Satya Nagaraja, Vishali Mogallapu, Dane Snow, Rich McCauley, Mustansir Mukadam, Iskender Agi, Shaun McCarthy, Zhanping Xu, Travis Perry, William Qian, Vei-Han Chan, Prabhu Adepu, Gazi Ali, Muneeb Ahmed, Aditya Mukherjee, Sheethal Nayak, Dave Gampell, Sunil Acharya, Lou Kordus, and Pat O'Connor. 2018. IMpixel 65nm BSI 320MHz demodulated TOF Image sensor with 3um global shutter pixels and analog binning. In 2018 IEEE International Solid - State Circuits Conference - (ISSCC). 94--96. 8310200
[4]
Google. 2021. Draco. https://github.com/google/draco.
[5]
Google. 2021. Glass. https://www.google.com/glass/start/.
[6]
GSMARENA. 2021. Google Pixel 3 XL. https://www.gsmarena.com/google_pixel_3_xl-9257.php.
[7]
Kiryong Ha, Zhuo Chen, Wenlu Hu, Wolfgang Richter, Padmanabhan Pillai, and Mahadev Satyanarayanan. 2014. Towards Wearable Cognitive Assistance. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '14). ACM.
[8]
Timo Hackel, Nikolay Savinov, Lubor Ladicky, Jan DirkWegner, Konrad Schindler, and Marc Pollefeys. 2017. Semantic3D.net: A new Large-scale Point Cloud Classification Benchmark. CoRR abs/1704.03847 (2017). arXiv:1704.03847 http: //arxiv.org/abs/1704.03847
[9]
Bo Han, Yu Liu, and Feng Qian. 2020. ViVo: Visibility-Aware Mobile Volumetric Video Streaming. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (London, United Kingdom) (MobiCom '20). Association for Computing Machinery, New York, NY, USA, Article 11, 13 pages.
[10]
Lei Han, Tian Zheng, Yinheng Zhu, Lan Xu, and Lu Fang. 2020. Live Semantic 3D Perception for Immersive Augmented Reality. IEEE Transactions on Visualization and Computer Graphics 26, 5 (2020), 2012--2022. 2020.2973477
[11]
Chris Hoffman and Craig Lloyd. 2018. The Best (Actually Useful) Tech We Saw at CES 2018. https://www.howtogeek.com/339206/the-best-actually-useful-tech-we-saw-at-ces-2018/.
[12]
Mohammad Hosseini and Christian Timmerer. 2018. Dynamic Adaptive Point Cloud Streaming. Proceedings of the 23rd Packet Video Workshop (Jun 2018).
[13]
Jinhan Hu, Alexander Shearer, Saranya Rajagopalan, and Robert LiKamWa. 2019. Banner: An Image Sensor Reconfiguration Framework for Seamless Resolution-Based Tradeoffs. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services (Seoul, Republic of Korea) (MobiSys '19). Association for Computing Machinery, New York, NY, USA, 236--248. https
[14]
Jinhan Hu, Jianan Yang, Vraj Delhivala, and Robert LiKamWa. 2018. Characterizing the Reconfiguration Latency of Image Sensor Resolution on Android Devices. In Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications (Tempe, Arizona, USA) (HotMobile '18). Association for Computing Machinery, New York, NY, USA, 81--86. https
[15]
Intel. 2021. RealSense Depth Camera D435i. https://www.intelrealsense.com/depth-camera-d435i/.
[16]
James Carroll. 2020. Time of Flight sensors target high-speed 3D machine vision tasks. https://www.vision-systems.com/cameras-accessories/article/14175785/3d-machine-vision-time-of-flight-tof-sensors-and-high-speed-tasks.
[17]
Stefan Lederer, Christopher Müller, and Christian Timmerer. 2012. Dynamic Adaptive Streaming over HTTP Dataset (MMSys '12). Association for Computing Machinery, New York, NY, USA, 89--94.
[18]
Kyungjin Lee, Juheon Yi, Youngki Lee, Sunghyun Choi, and Young Min Kim. 2020. GROOT: A Real-Time Streaming System of High-Fidelity Volumetric Videos (MobiCom '20). Association for Computing Machinery, New York, NY, USA, Article 57, 14 pages.
[19]
Germán Leiva, Cuong Nguyen, Rubaiat Habib Kazi, and Paul Asente. 2020. Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--13.
[20]
libjpeg-turbo. 2021. libjpeg-turbo. https://libjpeg-turbo.org/.
[21]
Point Cloud Library. 2021. Point Cloud Library. https://pointclouds.org/.
[22]
R. LiKamWa, J. Hu, V. Kodukula, and Y. Liu. 2021. Adaptive Resolution-Based Tradeoffs for Energy-Efficient Visual Computing Systems. IEEE Pervasive Computing (2021), 1--9.
[23]
Robert Likamwa, Jinhan Hu, Venkatesh Kodukula, and Yifei Liu. 2021. Adaptive Resolution-Based Tradeoffs for Energy-Efficient Visual Computing Systems. IEEE Pervasive Computing 20, 2 (2021), 18--26. 3052528
[24]
Robert LiKamWa, Bodhi Priyantha, Matthai Philipose, Lin Zhong, and Paramvir Bahl. 2013. Energy Characterization and Optimization of Image Sensing Toward Continuous Mobile Vision. In Proceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '13). ACM.
[25]
Zhi Liu, Qiyue Li, Xianfu Chen, Celimuge Wu, susumu ishihara, Jie Li, and Yusheng Ji. 2020. Point Cloud Video Streaming in 5G Systems and Beyond: Challenges and Solutions.
[26]
Microsoft. 2021. HoloLens 2. https://www.microsoft.com/en-us/hololens.
[27]
Microsoft Azure. 2021. Azure Kinect DK. https://azure.microsoft.com/en-us/services/kinect-dk/.
[28]
NVDIA DEVELOPER. 2021. nvJPEG Libraries. https://developer.nvidia.com/nvjpeg.
[29]
NVIDIA DEVELOPER. 2021. Jetson TX2 Module. https://developer.nvidia.com/embedded/jetson-tx2.
[30]
OmniVision. 2021. OV9282. https://www.ovt.com/sensors/OV9282.
[31]
Fabio Poiesi, Alex Locher, Paul Chippendale, Erica Nocerino, Fabio Remondino, and Luc Van Gool. 2017. Cloud-Based Collaborative 3D Reconstruction Using Smartphones. In Proceedings of the 14th European Conference on Visual Media Production (CVMP 2017) (London, United Kingdom) (CVMP 2017). Association for Computing Machinery, New York, NY, USA, Article 1, 9 pages.
[32]
Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2016. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. CoRR abs/1612.00593 (2016). arXiv:1612.00593 http://arxiv.org/abs/1612.00593
[33]
Feng Qian, Bo Han, Jarrell Pair, and Vijay Gopalakrishnan. 2019. Toward Practical Volumetric Video Streaming on Commodity Smartphones. In Proceedings of the 20th InternationalWorkshop on Mobile Computing Systems and Applications (Santa Cruz, CA, USA) (HotMobile '19). Association for Computing Machinery, New York, NY, USA, 135--140.
[34]
Hang Qiu, Fawad Ahmad, Fan Bai, Marco Gruteser, and Ramesh Govindan. 2018. AVR: Augmented Vehicular Reality. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services (Munich, Germany) (MobiSys '18). Association for Computing Machinery, New York, NY, USA, 81--95.
[35]
Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. 2017. OctNet: Learning Deep 3D Representations at High Resolutions. arXiv:1611.05009 [cs.CV]
[36]
Traci Ruether. 2021. The Impact of 5G on Streaming. https://www.wowza.com/blog/the-impact-of-5g-on-streaming.
[37]
Sabbir Rangwala. 2020. The iPhone 12 - LiDAR At Your Fingertips. https://www.forbes.com/sites/sabbirrangwala/2020/11/12/the-iphone-12lidar-at-your-fingertips.
[38]
Shishir Subramanyam, Irene Viola, Alan Hanjalic, and Pablo Cesar. 2020. User Centered Adaptive Streaming of Dynamic Point Clouds with Low Complexity Tiling. In Proceedings of the 28th ACM International Conference on Multimedia (Seattle, WA, USA) (MM '20). Association for Computing Machinery, New York, NY, USA, 3669--3677.
[39]
Unity. 2021. The leading platform for creating interactive, real-time content. https://unity.com/.
[40]
Tai Wang, Xinge Zhu, and Dahua Lin. 2020. Reconfigurable Voxels: A New Representation for LiDAR-Based Point Clouds. arXiv:2004.02724 [cs.CV]
[41]
Jun Yi, Md Reazul Islam, Shivang Aggarwal, Dimitrios Koutsonikolas, Y. Charlie Hu, and Zhisheng Yan. 2020. An Analysis of Delay in Live 360° Video Streaming Systems. In Proceedings of the 28th ACM International Conference on Multimedia (Seattle, WA, USA) (MM '20). Association for Computing Machinery, New York, NY, USA, 982--990.
[42]
Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. 2018. Open3D: A Modern Library for 3D Data Processing. arXiv:1801.09847 (2018).
[43]
Yin Zhou and Oncel Tuzel. 2018. VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4490--4499.

Cited By

View all
  • (2023)A QoS-Aware 3D Point Cloud Streaming from Real Space for Interaction in Metaverse2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)10.1109/PerComWorkshops56833.2023.10150337(68-73)Online publication date: 13-Mar-2023
  • (2023)Point Cloud Streaming over HTTP/22023 6th International Conference on Advanced Communication Technologies and Networking (CommNet)10.1109/CommNet60167.2023.10365247(1-5)Online publication date: 11-Dec-2023
  • (2023)A Tutorial on Immersive Video Delivery: From Omnidirectional Video to HolographyIEEE Communications Surveys & Tutorials10.1109/COMST.2023.326325225:2(1336-1375)Online publication date: 1-Apr-2023

Index Terms

  1. Characterizing real-time dense point cloud capture and streaming on mobile devices

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    HotEdgeVideo '21: Proceedings of the 3rd ACM Workshop on Hot Topics in Video Analytics and Intelligent Edges
    October 2021
    42 pages
    ISBN:9781450387002
    DOI:10.1145/3477083
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 25 October 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. dense point cloud streaming prototype
    2. performance and energy characterization
    3. point cloud rendering on mobile devices

    Qualifiers

    • Research-article

    Conference

    ACM MobiCom '21
    Sponsor:

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)85
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 27 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)A QoS-Aware 3D Point Cloud Streaming from Real Space for Interaction in Metaverse2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)10.1109/PerComWorkshops56833.2023.10150337(68-73)Online publication date: 13-Mar-2023
    • (2023)Point Cloud Streaming over HTTP/22023 6th International Conference on Advanced Communication Technologies and Networking (CommNet)10.1109/CommNet60167.2023.10365247(1-5)Online publication date: 11-Dec-2023
    • (2023)A Tutorial on Immersive Video Delivery: From Omnidirectional Video to HolographyIEEE Communications Surveys & Tutorials10.1109/COMST.2023.326325225:2(1336-1375)Online publication date: 1-Apr-2023

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media