research-article

A real-time remote rendering system for interactive mobile graphics

Authors:

Klara Nahrstedt,

Roy CampbellAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 8, Issue 3s

Article No.: 46, Pages 1 - 20

https://doi.org/10.1145/2348816.2348825

Published: 16 October 2012 Publication History

Abstract

Mobile devices are gradually changing people's computing behaviors. However, due to the limitations of physical size and power consumption, they are not capable of delivering a 3D graphics rendering experience comparable to desktops. Many applications with intensive graphics rendering workloads are unable to run on mobile platforms directly. This issue can be addressed with the idea of remote rendering: the heavy 3D graphics rendering computation runs on a powerful server and the rendering results are transmitted to the mobile client for display. However, the simple remote rendering solution inevitably suffers from the large interaction latency caused by wireless networks, and is not acceptable for many applications that have very strict latency requirements.

In this article, we present an advanced low-latency remote rendering system that assists mobile devices to render interactive 3D graphics in real-time. Our design takes advantage of an image based rendering technique: 3D image warping, to synthesize the mobile display from the depth images generated on the server. The research indicates that the system can successfully reduce the interaction latency while maintaining the high rendering quality by generating multiple depth images at the carefully selected viewpoints. We study the problem of viewpoint selection, propose a real-time reference viewpoint prediction algorithm, and evaluate the algorithm performance with real-device experiments.

References

[1]

Azuma, R. T. 1997. A survey of augmented reality. Presence: Teleoper. Virtual Environ. 6, 4, 355--385.

Digital Library

[2]

Bao, P. and Gourlay, D. 2004. Remote walkthrough over mobile networks using 3-D image warping and streaming. Vis. Image Signal Processing, IEE Proceedings 151, 4, 329--336.

[3]

Baratto, R. A., Kim, L. N., and Nieh, J. 2005. Thinc: A virtual display architecture for thin-client computing. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (SOSP'05). ACM, New York, 277--290.

Digital Library

[4]

Beigbeder, T., Coughlan, R., Lusher, C., Plunkett, J., Agu, E., and Claypool, M. 2004. The effects of loss and latency on user performance in unreal tournament 2003. In Proceedings of NetGames'04. 144--151.

Digital Library

[5]

Boukerche, A. and Pazzi, R. W. N. 2006. Remote rendering and streaming of progressive panoramas for mobile devices. In Proceedings of the 14th Annual ACM International Conference on Multimedia (Multimedia'06). ACM, New York, 691--694.

Digital Library

[6]

Chang, C.-F. and Ger, S.-H. 2002. Enhancing 3D graphics on mobile devices by image-based rendering. In Proceedings of PCM'02. 1105--1111.

Digital Library

[7]

Chen, S. E. 1995. Quicktime Vr: An image-based approach to virtual environment navigation. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'95). ACM, New York, 29--38.

Digital Library

[8]

Duguet, F. and Drettakis, G. 2004. Flexible point-based rendering on mobile devices. IEEE Trans. Comput. Graph. Appl. 24, 4, 57--63.

Digital Library

[9]

Eisert, P. and Fechteler, P. 2008. Low delay streaming of computer graphics. In Proceedings of the 15th IEEE International Conference on Image Processing, (ICIP 2008). 2704--2707.

[10]

Engel, K., Westermann, R., and Ertl, T. 1999. Isosurface extraction techniques for web-based volume visualization. IEEE Visualiz.

Digital Library

[11]

Epic Games. 2007. Unreal tournament 3. www.unrealtournament.com/.

[12]

Karlton, P. Womack, P., and Leech, J. 2005. Opengl graphics with the x window system (version 1.4). http://www.opengl.org/documentation/specs/.

[13]

Kum, S.-U. and Mayer-Patel, K. 2005. Real-time multidepth stream compression. ACM Trans. Multimedia Comput. Commun. Appl. 1, 128--150.

Digital Library

[14]

Kurillo, G., Vasudevan, R., Lobaton, E., and Bajcsy, R. 2008. A framework for collaborative real-time 3D teleimmersion in a geographically distributed environment. In Proceedings of the 10th IEEE International Symposium on Multimedia (ISM'08). 111--118.

Digital Library

[15]

Lamberti, F. and Sanna, A. 2007. A streaming-based solution for remote visualization of 3D graphics on mobile devices. IEEE Trans. Vis. Comput. Graph. 13, 2, 247--260.

Digital Library

[16]

Levoy, M. 1995. Polygon-assisted jpeg and mpeg compression of synthetic images. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). ACM, New York, 21--28.

Digital Library

[17]

Lien, J.-M., Kurillo, G., and Bajcsy, R. 2007. Seleton-based data compression for multi-camera tele-immersion system. In Proceedings of the 3rd International Conference on Advances in Visual Computing (ISVC'07). Vol. 1, Springer-Verlag, Berlin, Heidelberg, 714--723.

Digital Library

[18]

Mark, W. R. 1999. Post-rendering 3D image warping: Visibility, reconstruction, and performance for depth-image warping. Ph.D. dissertation. Department of Computer Science. University of North Carolina at Chapel Hill.

Digital Library

[19]

Marquez, J., Domenech, J., Gil, J., and Pont, A. 2008. Exploring the benefits of caching and prefetching in the mobile web. In Proceedings of WCITD'08.

[20]

McMillan, L. 1997. An image-based approach to three-dimensional computer graphics. Ph.D. Dissertation. Department of Computer Science. University of North Carolina at Chapel Hill.

Digital Library

[21]

McMillan, L. and Bishop, G. 1995. Plenoptic modeling: an image-based rendering system. In Proceedings of the 22nd Annual Conference on Computer Graphics and Internetive Techniques (SIGGRAPH'95). 39--46.

Digital Library

[22]

Noimark, Y. and Cohen-Or, D. 2003. Streaming scenes to mpeg-4 video-enabled devices. IEEE Comput. Graph. Appl. 23, 58--64.

Digital Library

[23]

Ohazama, C. 1999. Opengl vizserver. White paper, Silicon Graphics, Inc.

[24]

Penta, S. K. and Narayanan, P. 2005. Compression of multiple depth maps for ibr. Vis. Comput. 21, 611--618. 10.1007/s00371-005-0337-8.

[25]

Redert, A., de Beeck, M. O., Fehn, C., IJsselsteijn, W., Pollefeys, M., Gool, L. J. V., Ofek, E., Sexton, I., and Surman, P. 2002. Attest: Advanced three-dimensional television system technologies. In Proceedings of 1st International Symposium on 3D Data Processing Visualization Transmission (3DPVT). IEEE Computer Society, 313--319.

[26]

Riva, O. and Kangasharju, J. 2008. Challenges and lessons in developing middleware on smart phones. IEEE Computer 41, 10, 77--85.

Digital Library

[27]

ScanView. 2003. A system for remote visualization of scanned 3D models. http://graphics.stanford.edu/software/scanview/.

[28]

Schmalstieg, D. 1997. The remote rendering pipeline - managing geometry and bandwidth in distributed virtual environments. Ph.D. dissertation, Institute of Computer Graphics and Algorithms, Vienna University of Technology, Favoritenstrasse 9-11/186, A-1040 Vienna, Austria.

[29]

Shade, J., Gortler, S., He, L., and Szeliski, R. 1998. Layered depth images. In Proceedings of the 28th Annual Conference on Computer Graphics and Internetive Techniques (SIGGRAPH'98). 231--242.

Digital Library

[30]

Shi, S., Hsu, C.-H., Nahrstedt, K., and Campbell, R. 2011a. Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming. In Proceedings of the 19th ACM International Conference on Multimedia (MM '11). ACM, New York, NY, USA, 103--112.

Digital Library

[31]

Shi, S., Nahrstedt, K., and Campbell, R. H. 2011b. Distortion over latency: Novel metric for measuring interactive performance in remote rendering systems. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME'11).

Digital Library

[32]

Singhal, S. and Zyda, M. 1999. Networked Virtual Environments: Design and Implementation. ACM Press/Addison-Wesley Publishing Co., New York, NY.

Digital Library

[33]

Smit, F. A., van Liere, R., Beck, S., and Fröhlich, B. 2009. An image-warping architecture for Vr: Low latency versus image quality. In Proceedings of the IEEE Symposium on Virtual Reality (VR). IEEE, 27--34.

[34]

Zhu, M., Mondet, S., Morin, G., Ooi, W. T., and Cheng, W. 2011. Towards peer-assisted rendering in networked virtual environments. In Proceedings of the 19th ACM International Conference on Multimedia (MM '11). ACM, New York, 183--192.

Digital Library

Cited By

Geringer SGeiselhart FBäuerle ADec DOdenthal OReina GRopinski TWeiskopf D(2024)mint: Integrating scientific visualizations into virtual realityJournal of Visualization10.1007/s12650-024-01011-y27:6(1143-1169)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s12650-024-01011-y
Xu YLi YXu WGu YWang YMa JQi Z(2024)gFlow: Distributed Real-Time Reverse Remote Rendering System ModelMultiMedia Modeling10.1007/978-981-96-2061-6_1(3-16)Online publication date: 31-Dec-2024
https://doi.org/10.1007/978-981-96-2061-6_1
Wu JGuan YMao QCui YGuo ZZhang XSchulzrinne HKohler EMaltz DMisra V(2023)ZGaming: Zero-Latency 3D Cloud Gaming by Image PredictionProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604819(710-723)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604819
Show More Cited By

Index Terms

A real-time remote rendering system for interactive mobile graphics
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision

Recommendations

Real-time remote rendering of 3D video for mobile devices
MM '09: Proceedings of the 17th ACM international conference on Multimedia

At the convergence of computer vision, graphics, and multimedia, the emerging 3D video technology promises immersive experiences in a truly seamless environment. However, the requirements of huge network bandwidth and computing resources make it still a ...
Building low-latency remote rendering systems for interactive 3D graphics rendering on mobile devices
MM '11: Proceedings of the 19th ACM international conference on Multimedia
The recent explosion of mobile devices is changing people's computing behaviors and more and more applications are ported to mobile platforms. However, some applications, such as 3D video tele-immersion and 3D video gaming that require intensive ...
A Framework for Interactive 3D Rendering on Mobile Devices

Although mobile devices have now become an important computing platform, however most of them still lack hardware accelerated graphics. Therefore, interactive 3D rendering on these devices is a difficult task. This paper describes different approaches ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 8, Issue 3s

Special section of best papers of ACM multimedia 2011, and special section on 3D mobile multimedia

September 2012

173 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/2348816

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2012

Accepted: 01 June 2012

Revised: 01 May 2012

Received: 01 January 2012

Published in TOMM Volume 8, Issue 3s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Division of Computer and Network Systems

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

42
Total Citations
View Citations
972
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)6

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Geringer SGeiselhart FBäuerle ADec DOdenthal OReina GRopinski TWeiskopf D(2024)mint: Integrating scientific visualizations into virtual realityJournal of Visualization10.1007/s12650-024-01011-y27:6(1143-1169)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s12650-024-01011-y
Xu YLi YXu WGu YWang YMa JQi Z(2024)gFlow: Distributed Real-Time Reverse Remote Rendering System ModelMultiMedia Modeling10.1007/978-981-96-2061-6_1(3-16)Online publication date: 31-Dec-2024
https://doi.org/10.1007/978-981-96-2061-6_1
Wu JGuan YMao QCui YGuo ZZhang XSchulzrinne HKohler EMaltz DMisra V(2023)ZGaming: Zero-Latency 3D Cloud Gaming by Image PredictionProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604819(710-723)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604819
Illahi GVaishnav AKämäräinen TSiekkinen MDi Francesco MZimmermann RShirmohammadi SGriwodz CHefeeda MWang M(2023)Learning to Predict Head Pose in Remotely-Rendered Virtual RealityProceedings of the 14th Conference on ACM Multimedia Systems10.1145/3587819.3590972(27-38)Online publication date: 7-Jun-2023
https://dl.acm.org/doi/10.1145/3587819.3590972
ma cli lwang yliu yxu s(2023)AR-assisted intelligent analysis and identification system for mobile vegetables diseases based on HOG-SVMInternational Conference on Artificial Intelligence, Virtual Reality, and Visualization (AIVRV 2022)10.1117/12.2668866(60)Online publication date: 1-Mar-2023
https://doi.org/10.1117/12.2668866
Lu EBharadwaj SDasari MSmith CSeshan SRowe A(2023)RenderFusion: Balancing Local and Remote Rendering for Interactive 3D Scenes2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00046(312-321)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00046
Dóka JNagy BFormanek BJocha DKern APéter Gerő BSonkoly B(2023)A Novel Split Rendering XR Framework with Occlusion Support2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00169(772-774)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00169
Kang JSim JBahn H(2023)Optimization of the Modified Gaussian Filter for Mobile GPU Usage in Game Workloads2023 International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI)10.1109/CCCI58712.2023.10290860(1-8)Online publication date: 18-Oct-2023
https://doi.org/10.1109/CCCI58712.2023.10290860
Wu HZhang HCheng JGuo JChen W(2023)Perspectives on point cloud-based 3D scene modeling and XR presentation within the cloud-edge-client architectureVisual Informatics10.1016/j.visinf.2023.06.0077:3(59-64)Online publication date: Sep-2023
https://doi.org/10.1016/j.visinf.2023.06.007
Kravec JKáčerik MBittner J(2023)PVLI: potentially visible layered image for real-time ray tracingThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-03007-539:8(3359-3372)Online publication date: 28-Jul-2023
https://dl.acm.org/doi/10.1007/s00371-023-03007-5
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents