research-article

A high-quality low-delay remote rendering system for 3D video

Authors:

Klara Nahrstedt,

Roy H. CampbellAuthors Info & Claims

MM '10: Proceedings of the 18th ACM international conference on Multimedia

Pages 601 - 610

https://doi.org/10.1145/1873951.1874011

Published: 25 October 2010 Publication History

Abstract

As an emerging technology, 3D video has shown a great potential to become the next generation media for tele-immersion. However, streaming and rendering this dynamic 3D data in real-time requires tremendous network bandwidth and computing resources. In this paper, we build a remote rendering model to better study different remote rendering designs and define 3D video rendering as an optimization problem. Moreover, we design a 3D video remote rendering system that significantly reduces the delay while maintaining high rendering quality. We also propose a reference viewpoint prediction algorithm with super sampling support that requires much less computation resources but provides better performance than the search-based algorithms proposed in the related work.

References

[1]

Bumblebee 2 camera. http://www.ptgrey.com/products/bumblebee2/.

[2]

Teeve project. http://cairo.cs.uiuc.edu/teleimmersion/.

[3]

P. Bao and D. Gourlay. Remote walkthrough over mobile networks using 3-D image warping and streaming. Vision, Image and Signal Processing, IEEE Proceedings, 151(4):329--336, Aug. 2004.

[4]

A. Boukerche and R. Pazzi. Remote rendering and streaming of progressive panoramas for mobile devices. In Multimedia '06, pages 691--694. ACM, 2006.

Digital Library

[5]

C.-F. Chang and S.-H. Ger. Enhancing 3D graphics on mobile devices by image-based rendering. In PCM '02, pages 1105--1111, London, UK, 2002.

Digital Library

[6]

S. E. Chen. Quicktime vr: an image-based approach to virtual environment navigation. In SIGGRAPH '95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 29--38, New York, NY, USA, 1995. ACM.

Digital Library

[7]

S. E. Chen and L. Williams. View interpolation for image synthesis. In SIGGRAPH '93: Proceedings of the 20th annual conference on Computer graphics and interactive techniques, pages 279--288, 1993.

Digital Library

[8]

M. Claypool and K. Claypool. Latency can kill: precision and deadline in online games. In MMSys '10: Proceedings of the first annual ACM SIGMM conference on Multimedia systems, pages 215--222, New York, NY, USA, 2010. ACM.

Digital Library

[9]

F. Duguet and G. Drettakis. Flexible point-based rendering on mobile devices. IEEE Trans. Computer Graphics and Applications, 24(4):57--63, 2004.

Digital Library

[10]

K. Engel, R. Westermann, and T. Ertl. Isosurface extraction techniques for web-based volume visualization. In IEEE Visualization, 1999.

Digital Library

[11]

H. Kim, I. Kitahara, R. Sakamoto, and K. Kogure. An immersive free-viewpoint video system using multiple outer/inner cameras. In 3DPVT, pages 782--789, 2006.

Digital Library

[12]

G. Kurillo, R. Vasudevan, E. Lobaton, and R. Bajcsy. A framework for collaborative real-time 3D teleimmersion in a geographically distributed environment. In ISM, pages 111--118, 2008.

Digital Library

[13]

F. Lamberti and A. Sanna. A streaming-based solution for remote visualization of 3D graphics on mobile devices. IEEE Trans. Vis. Comput. Graph., 13(2):247--260, 2007.

Digital Library

[14]

W. R. Mark. Post-rendering 3D image warping: Visibility, reconstruction, and performance for depth-image warping. In Ph.D. Dissertation. University of North Carolina at Chapel Hill, Department of Computer Science, 1999.

Digital Library

[15]

L. McMillan. An image-based approach to three-dimensional computer graphics. In Ph.D. Dissertation. University of North Carolina at Chapel Hill, Department of Computer Science, 1997.

Digital Library

[16]

H. Mobahi, S. R. Rao, and Y. Ma. Data-driven image completion by image patch subspaces. In PCS'09: Proceedings of the 27th conference on Picture Coding Symposium, pages 241--244, Piscataway, NJ, 2009.

Digital Library

[17]

S. Prohaska, A. Hutanu, R. Kahler, and H.-C. Hege. Interactive exploration of large remote micro-ct scans. In VIS '04: Proceedings of the conference on Visualization '04, pages 345--352, Washington, DC, USA, 2004. IEEE Computer Society.

Digital Library

[18]

A. Redert, M. O. de Beeck, C. Fehn, W. IJsselsteijn, M. Pollefeys, L. J. V. Gool, E. Ofek, I. Sexton, and P. Surman. Attest: Advanced three-dimensional television system technologies. In 3DPVT, pages 313--319. IEEE Computer Society, 2002.

[19]

J. Shade, S. Gortler, L. He, and R. Szeliski. Layered depth images. In SIGGRAPH, pages 231--242, 1998.

Digital Library

[20]

S. Shi, W. J. Jeon, K. Nahrstedt, and R. H. Campbell. Real-time remote rendering of 3D video for mobile devices. In MM '09: Proceedings of the seventeen ACM international conference on Multimedia, pages 391--400, New York, NY, USA, 2009. ACM.

Digital Library

[21]

A. Smolic, K. M¨uller, P. Merkle, C. Fehn, P. Kauff, P. Eisert, and T. Wiegand. 3D video and free viewpoint video - technologies, applications and mpeg standards. In ICME, pages 2161--2164. IEEE, 2006.

[22]

S. Wurmlin, E. Lamboray, O. G. Staadt, and M. H. Gross. 3D video recorder: a system for recording and playing free-viewpoint video. Comput. Graph. Forum, 22(2):181--194, 2003.

[23]

Z. Yang, K. Nahrstedt, Y. Cui, B. Yu, J. Liang, S.-H. Jung, and R. Bajcsy. Teeve: The next generation architecture for tele-immersive environment. In ISM, pages 112--119. IEEE Computer Society, 2005.

Digital Library

Cited By

Wu JGuan YMao QCui YGuo ZZhang XSchulzrinne HKohler EMaltz DMisra V(2023)ZGaming: Zero-Latency 3D Cloud Gaming by Image PredictionProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604819(710-723)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604819
Wang RZhang BWu MZhang JGuo XZhang XLi HJiao DMa S(2019)Performance bottleneck analysis and resource optimized distribution method for IoT cloud rendering computing system in cyber-enabled applicationsEURASIP Journal on Wireless Communications and Networking10.1186/s13638-019-1401-92019:1Online publication date: 29-Mar-2019
https://doi.org/10.1186/s13638-019-1401-9
Wang XLiang XYang BLi F(2018)Scalable Remote Rendering Using Synthesized Image Quality AssessmentIEEE Access10.1109/ACCESS.2018.28531326(36595-36610)Online publication date: 2018
https://doi.org/10.1109/ACCESS.2018.2853132
Show More Cited By

Index Terms

A high-quality low-delay remote rendering system for 3D video
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision

Recommendations

Real-time remote rendering of 3D video for mobile devices
MM '09: Proceedings of the 17th ACM international conference on Multimedia

At the convergence of computer vision, graphics, and multimedia, the emerging 3D video technology promises immersive experiences in a truly seamless environment. However, the requirements of huge network bandwidth and computing resources make it still a ...
A real-time remote rendering system for interactive mobile graphics
Special section of best papers of ACM multimedia 2011, and special section on 3D mobile multimedia

Mobile devices are gradually changing people's computing behaviors. However, due to the limitations of physical size and power consumption, they are not capable of delivering a 3D graphics rendering experience comparable to desktops. Many applications ...
Building low-latency remote rendering systems for interactive 3D graphics rendering on mobile devices
MM '11: Proceedings of the 19th ACM international conference on Multimedia
The recent explosion of mobile devices is changing people's computing behaviors and more and more applications are ported to mobile platforms. However, some applications, such as 3D video tele-immersion and 3D video gaming that require intensive ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '10: Proceedings of the 18th ACM international conference on Multimedia

October 2010

1836 pages

ISBN:9781605589336

DOI:10.1145/1873951

General Chairs:
Alberto del Bimbo
University of Florence, Italy
,
Shih-Fu Chang
Columbia University, USA
,
Program Chair:
Arnold Smeulders
University of Amsterdam, NL

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 October 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '10

Sponsor:

SIGMM

MM '10: ACM Multimedia Conference

October 25 - 29, 2010

Firenze, Italy

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
524
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu JGuan YMao QCui YGuo ZZhang XSchulzrinne HKohler EMaltz DMisra V(2023)ZGaming: Zero-Latency 3D Cloud Gaming by Image PredictionProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604819(710-723)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604819
Wang RZhang BWu MZhang JGuo XZhang XLi HJiao DMa S(2019)Performance bottleneck analysis and resource optimized distribution method for IoT cloud rendering computing system in cyber-enabled applicationsEURASIP Journal on Wireless Communications and Networking10.1186/s13638-019-1401-92019:1Online publication date: 29-Mar-2019
https://doi.org/10.1186/s13638-019-1401-9
Wang XLiang XYang BLi F(2018)Scalable Remote Rendering Using Synthesized Image Quality AssessmentIEEE Access10.1109/ACCESS.2018.28531326(36595-36610)Online publication date: 2018
https://doi.org/10.1109/ACCESS.2018.2853132
Zhu MMorin GCharvillat VOoi W(2017)Sprite treeThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-016-1286-033:11(1385-1402)Online publication date: 1-Nov-2017
https://dl.acm.org/doi/10.1007/s00371-016-1286-0
Shi SHsu C(2015)A Survey of Interactive Remote Rendering SystemsACM Computing Surveys10.1145/271992147:4(1-29)Online publication date: 26-May-2015
https://dl.acm.org/doi/10.1145/2719921
(2014)Cloud-Assisted Smart Camera Networks for Energy-Efficient 3D Video StreamingComputer10.1109/MC.2014.11447:5(60-66)Online publication date: May-2014
https://doi.org/10.1109/MC.2014.114
Maugey TFrossard P(2013)Interactive Multiview Video System With Low Complexity 2D Look Around at DecoderIEEE Transactions on Multimedia10.1109/TMM.2013.224614715:5(1070-1082)Online publication date: Aug-2013
https://doi.org/10.1109/TMM.2013.2246147
Kapetanakis KPanagiotakis S(2012)Evaluation of techniques for web 3D graphics animation on portable devices2012 International Conference on Telecommunications and Multimedia (TEMU)10.1109/TEMU.2012.6294708(152-157)Online publication date: Jul-2012
https://doi.org/10.1109/TEMU.2012.6294708
Zhu MMondet SMorin GOoi WCheng WCandan KPanchanathan SPrabhakaran BSundaram HFeng WSebe N(2011)Towards peer-assisted rendering in networked virtual environmentsProceedings of the 19th ACM international conference on Multimedia10.1145/2072298.2072324(183-192)Online publication date: 28-Nov-2011
https://dl.acm.org/doi/10.1145/2072298.2072324
Shi SHsu CNahrstedt KCampbell RCandan KPanchanathan SPrabhakaran BSundaram HFeng WSebe N(2011)Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gamingProceedings of the 19th ACM international conference on Multimedia10.1145/2072298.2072313(103-112)Online publication date: 28-Nov-2011
https://dl.acm.org/doi/10.1145/2072298.2072313
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten