Multi-frame Bitrate Allocation of Dynamic 3D Gaussian Splatting Streaming Over Dynamic Networks
Authors: 
Yuan-Chun Sun, 
Yuang Shi, 
Wei Tsang Ooi, 
Chun-Ying Huang, Cheng-Hsin HsuAuthors Info & Claims
Yuan-Chun Sun, Yuang Shi, Wei Tsang Ooi, Chun-Ying Huang, Cheng-Hsin HsuAuthors Info & ClaimsPages 1 - 7
Abstract
Dynamic 3D Gaussian splats have emerged as an exciting new data type for modeling interactive photo-realistic 3D scenes. This work considers the problem of bitrate allocation for streaming dynamic 3D Gaussian splats under dynamic network conditions. We model four parameters that influence the rate-distortion trade-offs for different attribute categories and propose an efficient Model-driven Gradient Ascent (MGA) algorithm to search for the optimal parameters that achieve high visual quality while keeping the bitrate below a given threshold across multiple frames. In our experiments, MGA achieves up to 5.46 dB in PSNR improvement over the baseline. We further proposed an adaptive MGA that reduces close to 3x computational time with negligible visual quality loss.
References
[1]
Yihang Chen, Qianyi Wu, Jianfei Cai, Mehrtash Harandi, and Weiyao Lin. 2024. HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression. arXiv preprint arXiv:2403.14530 (2024).
[2]
Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. 2024. LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS. arXiv preprint arXiv:2311.17245 (2024).
[3]
Guangchi Fang and Bing Wang. 2024. Mini-Splatting: Representing Scenes with a Constrained Number of Gaussians. arXiv preprint arXiv:2403.14166 (2024).
[4]
Sharath Girish, Kamal Gupta, and Abhinav Shrivastava. 2024. EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS. arXiv preprint arXiv:2312.04564 (2024).
[5]
Google. 2017. Draco 3D Data Compression. https://github.com/google/draco.
[6]
Alain Horé and Djemel Ziou. 2010. Image Quality Metrics: PSNR vs. SSIM. In 2010 20th International Conference on Pattern Recognition (ICPR). Istanbul, Turkey, 2366--2369.
[7]
Kai Katsumata, Duc Minh Vo, and Hideki Nakayama. 2023. An Efficient 3D Gaussian Representation for Monocular/Multi-view Dynamic Scenes. arXiv preprint arXiv:2311.12897 (2023).
[8]
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023. 3D Gaussian Splatting for Real-time Radiance Field Rendering. ACM Transactions on Graphics 42, 4 (2023), 1--14.
[9]
Wenkai Liu, Tao Guan, Bin Zhu, Lili Ju, Zikai Song, Dan Li, Yuesong Wang, and Wei Yang. 2024. EfficientGS: Streamlining Gaussian Splatting for Large-Scale High-Resolution Scene Representation. arXiv preprint arXiv:2404.12777 (2024).
[10]
Xiangrui Liu, Xinju Wu, Pingping Zhang, Shiqi Wang, Zhu Li, and Sam Kwong. 2024. CompGS: Efficient 3D Scene Representation via Compressed Gaussian Splatting. arXiv preprint arXiv:2404.09458 (2024).
[11]
Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. 2024. Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA, 20654--20664.
[12]
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. 2023. Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis. arXiv preprint arXiv:2310.08528 (2023).
[13]
Haimin Luo, Teng Xu, Yuheng Jiang, Chenglin Zhou, Qiwei Qiu, Yingliang Zhang, Wei Yang, Lan Xu, and Jingyi Yu. 2022. Artemis: Articulated Neural Pets with Appearance and Motion Synthesis. ACM Transactions on Graphics 41, 4 (July 2022), 1--19.
[14]
MPEG. 2021. G-PCC codec description v12. Technical Report. ISO/IEC JTC 1/SC 29/WG 7 MPEG Output Document.
[15]
Arvind Narayanan, Eman Ramadan, Rishabh Mehta, Xinyue Hu, Qingxu Liu, Rostand A. K. Fezeu, Udhaya Kumar Dayalan, Saurabh Verma, Peiqi Ji, Tao Li, Feng Qian, and Zhi-Li Zhang. 2020. Lumos5G: Mapping and Predicting Commercial mmWave 5G Throughput. In Proceedings of the ACM Internet Measurement Conference (IMC). Virtual Event, USA, 176--193.
[16]
Netflix. 2021. VMAF - Video Multi-Method Assessment Fusion. https://github.com/Netflix/vmaf.
[17]
Simon Niedermayr, Josef Stumpfegger, and Rüdiger Westermann. 2024. Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA, 10349--10358.
[18]
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (Nov. 2011), 2825--2830.
[19]
Aras Pranckevicius. 2023. Making Gaussian Splats Smaller. https://aras-p.info/blog/2023/09/13/Making-Gaussian-Splats-smaller/.
[20]
Kerui Ren, Lihan Jiang, Tao Lu, Mulin Yu, Linning Xu, Zhangkai Ni, and Bo Dai. 2024. Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians. arXiv preprint arXiv:2403.17898 (2024).
[21]
Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 2024. 3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA, 20675--20685.
[22]
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 2023. 4D Gaussian Splatting for Real-time Dynamic Scene Rendering. arXiv preprint arXiv:2310.08528 (2023).
[23]
Zerong Zheng, Han Huang, Tao Yu, Hongwen Zhang, Yandong Guo, and Yebin Liu. 2022. Structured Local Radiance Fields for Human Avatar Modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA, 15893--15903.
Index Terms
- Multi-frame Bitrate Allocation of Dynamic 3D Gaussian Splatting Streaming Over Dynamic Networks
Recommendations
PRTGS: Precomputed Radiance Transfer of Gaussian Splats for Real-Time High-Quality Relighting
MM '24: Proceedings of the 32nd ACM International Conference on MultimediaWe proposed Precomputed Radiance Transfer of Gaussian Splats (PRTGS), a real-time high-quality relighting method for Gaussian splats in low-frequency lighting environments that captures soft shadows and interreflections by precomputing 3D Gaussian splats'...
Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections
Computer Vision – ECCV 2024AbstractNovel view synthesis from unconstrained in-the-wild images remains a meaningful but challenging task. The photometric variation and transient occluders in those unconstrained images make it difficult to reconstruct the original scene accurately. ...
SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting
Computer Vision – ECCV 2024AbstractNovel view synthesis has shown rapid progress recently, with methods capable of producing increasingly photorealistic results. 3D Gaussian Splatting has emerged as a promising method, producing high-quality renderings of scenes and enabling ...
Comments
Information & Contributors
Information
Published In
August 2024
63 pages
ISBN:9798400707117
DOI:10.1145/3672196
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ACM SIGCOMM '24
Sponsor:
Acceptance Rates
EMS '24 Paper Acceptance Rate 9 of 15 submissions, 60%;
Overall Acceptance Rate 9 of 15 submissions, 60%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 391Total Downloads
- Downloads (Last 12 months)391
- Downloads (Last 6 weeks)30
Reflects downloads up to 27 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in