Jiangkai Wu
ABSTRACT
In cloud gaming, interactive latency is one of the most important factors in users' experience. Although the interactive latency can be reduced through typical network infrastructures like edge caching and congestion control, the interactive latency of current cloud-gaming platforms is still far from users' satisfaction.
This paper presents ZGaming, a novel 3D cloud gaming system based on image prediction, in order to eliminate the interactive latency in traditional cloud gaming systems. To improve the quality of the predicted images, we propose (1) a quality-driven 3D-block cache to reduce the "hole" artifacts, (2) a server-assisted LSTM-predicting algorithm to improve the prediction accuracy of dynamic foreground objects, and (3) a prediction-performance-driven adaptive bitrate strategy which optimizes the quality of predicted images. The experiment on the real-world cloud gaming network conditions shows that compared with existing methods, ZGaming reduces the interactive latency from 23 ms to 0 ms when providing the same video quality, or improves the video quality by 5.4 dB when keeping the interactive latency as 0 ms.
- 2013. X264. (2013). https://www.videolan.org/developers/x264.html.Google Scholar
- 2018. gamehook. (2018). https://github.com/philkr/gamehook.Google Scholar
- 2020. GTA-IM-Dataset. (2020). https://github.com/ZheC/GTA-IM-Dataset.Google Scholar
- 2022. Amazon Luna. (2022). https://www.amazon.com/luna/landing-page.Google Scholar
- 2022. Cyberpunk 2077. (2022). https://www.cyberpunk.net/.Google Scholar
- 2022. DirectX. (2022). https://en.wikipedia.org/wiki/DirectX.Google Scholar
- 2022. Google Stadia. (2022). https://stadia.google.com.Google Scholar
- 2022. gpu. (2022). https://developer.nvidia.com/cuda-gpus.Google Scholar
- 2022. Microsoft Xbox Cloud Gaming. (2022). https://www.xbox.com/en-US/xbox-game-pass/cloud-gaming.Google Scholar
- 2022. NVIDIA GeForce Now. (2022). https://www.nvidia.com/en-us/geforce-now/.Google Scholar
- 2022. opengl. (2022). https://www.opengl.org/.Google Scholar
- 2022. Razor. (2022). https://github.com/yuanrongxi/razor.Google Scholar
- 2022. RenderDoc. (2022). https://renderdoc.org/.Google Scholar
- 2022. Sony PlayStation Now. (2022). https://www.playstation.com/en-us/ps-now/.Google Scholar
- 2022. ue. (2022). https://www.unrealengine.com/.Google Scholar
- 2022. unity. (2022). https://unity.com/.Google Scholar
- 2023. Cloud Gaming for Genshin Impact. (2023). https://mhyy.mihoyo.com.Google Scholar
- 2023. Genshin Impact. (2023). https://genshin.hoyoverse.com/.Google Scholar
- 2023. NetLimiter. (2023). https://www.netlimiter.com.Google Scholar
- 2023. Splatoon. (2023). https://splatoon.nintendo.com.Google Scholar
- Alberto Alós, Francisco Morán, Pablo Carballeira, Daniel Berjón, and Narciso García. 2019. Congestion Control for Cloud Gaming Over UDP Based on Round-Trip Video Latency. IEEE Access 7 (2019), 78882--78897. Google ScholarCross Ref
- Maryam Amiri, Hussein Al Osman, Shervin Shirmohammadi, and Maha Abdallah. 2015. An SDN controller for delay and jitter reduction in cloud gaming. In Proceedings of the 23rd ACM international conference on Multimedia. 1043--1046.Google ScholarDigital Library
- Maryam Amiri, Hussein Al Osman, Shervin Shirmohammadi, and Maha Abdallah. 2016. Toward delay-efficient game-aware data centers for cloud gaming. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 12, 5s (2016), 1--19.Google ScholarDigital Library
- Paul Bao and Douglas Gourlay. 2004. Remote walkthrough over mobile networks using 3-D image warping and streaming. IEE Proceedings-Vision, Image and Signal Processing 151, 4 (2004), 329--336.Google ScholarCross Ref
- Neal Cardwell, Yuchung Cheng, C Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. 2017. BBR: congestion-based congestion control. Commun. ACM 60, 2 (2017), 58--66.Google ScholarDigital Library
- Gaetano Carlucci, Luca De Cicco, Stefan Holmer, and Saverio Mascolo. 2016. Analysis and design of the google congestion control for web real-time communication (WebRTC). In Proceedings of the 7th International Conference on Multimedia Systems. 1--12.Google ScholarDigital Library
- Marc Carrascosa and Boris Bellalta. 2020. Cloud-gaming:Analysis of Google Stadia traffic. (2020). arXiv:cs.NI/2009.09786Google Scholar
- Chun-Fa Chang and Shyh-Haur Ger. 2002. Enhancing 3D graphics on mobile devices by image-based rendering. In Pacific-Rim Conference on Multimedia. Springer, 1105--1111.Google ScholarCross Ref
- Mark Claypool and Kajal Claypool. 2006. Latency and player actions in online games. Commun. ACM 49, 11 (2006), 40--45.Google ScholarDigital Library
- Mark Claypool and Kajal Claypool. 2010. Latency can kill: precision and deadline in online games. In Proceedings of the first annual ACM SIGMM conference on Multimedia systems. 215--222.Google ScholarDigital Library
- Tongyu Dai, Xinggong Zhang, and Zongming Guo. 2018. Learning-based congestion control for internet video communication over wireless networks. In 2018 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 1--5.Google ScholarCross Ref
- Andrea Di Domenico, Gianluca Perna, Martino Trevisan, Luca Vassio, and Danilo Giordano. 2021. A network analysis on cloud gaming: Stadia, GeForce Now and PSNow. (2021). arXiv:cs.NI/2012.06774Google Scholar
- Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2015. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence 38, 2 (2015), 295--307.Google Scholar
- Christoph Fehn. 2004. Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In Stereoscopic Displays and Virtual Reality Systems XI, Vol. 5291. International Society for Optics and Photonics, 93--104.Google ScholarCross Ref
- Chun-Ying Huang, Kuan-Ta Chen, De-Yu Chen, Hwai-Jung Hsu, and Cheng-Hsin Hsu. 2014. GamingAnywhere: The first open source cloud gaming system. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 10, 1s (2014), 1--25.Google ScholarDigital Library
- Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. 2018. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2821--2830.Google ScholarCross Ref
- Vincent Jantet, Christine Guillemot, and Luce Morin. 2011. Object-based layered depth images for improved virtual view synthesis in rate-constrained context. In 2011 18th IEEE International Conference on Image Processing. IEEE, 125--128.Google ScholarCross Ref
- Jian Jin, Anhong Wang, Yao Zhao, Chunyu Lin, and Bing Zeng. 2016. Region-aware 3-D warping for DIBR. IEEE Transactions on Multimedia 18, 6 (2016), 953--966.Google ScholarDigital Library
- Philipp Krähenbühl. 2018. Free supervision from video games. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2955--2964.Google ScholarCross Ref
- Zeqi Lai, Y. Charlie Hu, Yong Cui, Linhui Sun, Ningwei Dai, and Hung-Sheng Lee. 2020. Furion: Engineering High-Quality Immersive Virtual Reality on Today's Mobile Devices. IEEE Transactions on Mobile Computing 19, 7 (2020), 1586--1602. Google ScholarCross Ref
- Ji-Tang Lee, De-Nian Yang, and Wanjiun Liao. 2016. Efficient caching for multi-view 3D videos. In 2016 IEEE Global Communications Conference (GLOBECOM). IEEE, 1--7.Google ScholarDigital Library
- Kyungmin Lee, David Chu, Eduardo Cuervo, Johannes Kopf, Yury Degtyarev, Sergey Grizan, Alec Wolman, and Jason Flinn. 2015. Outatime: Using Speculation to Enable Low-Latency Continuous Interaction for Mobile Cloud Gaming. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '15). Association for Computing Machinery, New York, NY, USA, 151--165. Google ScholarDigital Library
- Shuai Li, Kaixin Wang, Yanbo Gao, Xun Cai, and Mao Ye. 2022. Geometric warping error aware CNN for DIBR oriented view synthesis. In Proceedings of the 30th ACM International Conference on Multimedia. 1512--1521.Google ScholarDigital Library
- Yuhua Lin and Haiying Shen. 2017. CloudFog: Leveraging Fog to Extend Cloud Gaming for Thin-Client MMOG with High Quality of Service. IEEE Transactions on Parallel and Distributed Systems 28, 2 (2017), 431--445. Google ScholarDigital Library
- Liying Lu, Wenbo Li, Xin Tao, Jiangbo Lu, and Jiaya Jia. 2021. Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6368--6377.Google ScholarCross Ref
- William R Mark. 1999. Post-rendering 3D image warping: visibility, reconstruction, and performance for depth-image warping. Ph.D. Dissertation. Citeseer.Google Scholar
- Leonard McMillan. 1997. An image-based approach to three-dimensional computer graphics. Ph.D. Dissertation. Citeseer.Google ScholarDigital Library
- Voicu Popescu, John Eyles, Anselmo Lastra, Joshua Steinhurst, Nick England, and Lars Nyland. 2000. The WarpEngine: An architecture for the post-polygonal age. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 433--442.Google ScholarDigital Library
- Xiaojuan Qi, Zhengzhe Liu, Qifeng Chen, and Jiaya Jia. 2019. 3D motion decomposition for RGBD future dynamic scene synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7673--7682.Google ScholarCross Ref
- Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. 2019. On the spectral bias of neural networks. In International Conference on Machine Learning. PMLR, 5301--5310.Google Scholar
- Konstantinos Rematas, Ira Kemelmacher-Shlizerman, Brian Curless, and Steve Seitz. 2018. Soccer on your tabletop. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4738--4747.Google ScholarCross Ref
- Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. 2016. Playing for data: Ground truth from computer games. In European conference on computer vision. Springer, 102--118.Google ScholarCross Ref
- Saeed Shafiee Sabet, Steven Schmidt, Saman Zadtootaghaj, Babak Naderi, Carsten Griwodz, and Sebastian Möller. 2020. A latency compensation technique based on game characteristics to mitigate the influence of delay on cloud gaming quality of experience. In Proceedings of the 11th ACM Multimedia Systems Conference. 15--25.Google ScholarDigital Library
- Michael Schaffner, Pierre Greisen, Simon Heinzle, Frank K Gürkaynak, Hubert Kaeslin, and Aljoscha Smolic. 2013. MADmax: A 1080p stereo-to-multiview rendering ASIC in 65 nm CMOS based on image domain warping. In 2013 Proceedings of the ESSCIRC (ESSCIRC). IEEE, 61--64.Google ScholarCross Ref
- Shu Shi, Cheng-Hsin Hsu, Klara Nahrstedt, and Roy Campbell. 2011. Using graphics rendering contexts to enhance the real-time video coding for mobile cloud gaming. In Proceedings of the 19th ACM international conference on Multimedia. 103--112.Google ScholarDigital Library
- Shu Shi, Won J Jeon, Klara Nahrstedt, and Roy H Campbell. 2009. Real-time remote rendering of 3D video for mobile devices. In Proceedings of the 17th ACM international conference on Multimedia. 391--400.Google ScholarDigital Library
- Shu Shi, Mahsa Kamali, Klara Nahrstedt, John C Hart, and Roy H Campbell. 2010. A high-quality low-delay remote rendering system for 3D video. In Proceedings of the 18th ACM international conference on Multimedia. 601--610.Google ScholarDigital Library
- Shu Shi, Klara Nahrstedt, and Roy Campbell. 2012. A real-time remote rendering system for interactive mobile graphics. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 8, 3s (2012), 1--20.Google ScholarDigital Library
- Jiahao Su, Wonmin Byeon, Jean Kossaiff, Furong Huang, Jan Kautz, and Anima Anandkumar. 2020. Convolutional tensor-train lstm for spatio-temporal learning. Advances in Neural Information Processing Systems 33 (2020), 13714--13726.Google Scholar
- Mirko Suznjevic, Ivan Slivar, and Lea Skorin-Kapov. 2016. Analysis and QoE evaluation of cloud gaming service adaptation under different network conditions: The case of NVIDIA GeForce NOW. In 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX). 1--6. Google ScholarCross Ref
- Dong Tian, Po-Lin Lai, Patrick Lopez, and Cristina Gomila. 2009. View synthesis techniques for 3D video. In Applications of Digital Image Processing XXXII, Vol. 7443. SPIE, 233--243.Google Scholar
- Shishun Tian, Lu Zhang, Luce Morin, and Olivier Déforges. 2019. A Benchmark of DIBR Synthesized View Quality Assessment Metrics on a New Database for Immersive Media Applications. IEEE Transactions on Multimedia 21, 5 (2019), 1235--1247. Google ScholarDigital Library
- YunboWang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and S Yu Philip. 2018. Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In International Conference on Machine Learning. PMLR, 5123--5132.Google Scholar
- Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip Yu, and Mingsheng Long. 2022. Predrnn: A recurrent neural network for spatiotem-poral predictive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).Google Scholar
- Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip S Yu. 2019. Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9154--9162.Google ScholarCross Ref
- Lingfeng Xu, Xun Guo, Yan Lu, Shipeng Li, Oscar C. Au, and Lu Fang. 2014. A low latency cloud gaming system using edge preserved image homography. In 2014 IEEE International Conference on Multimedia and Expo (ICME). 1--6. Google ScholarCross Ref
- Roy D Yates, Mehrnaz Tavan, Yi Hu, and Dipankar Raychaudhuri. 2017. Timely cloud gaming. In IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 1--9.Google ScholarCross Ref
- Mengsi Yeh, Chih-Hang Wang, Jitang Lee, De-Nian Yang, and Wanjiun Liao. 2020. Mobile proxy caching for multi-view 3D videos with adaptive view selection. IEEE Transactions on Mobile Computing (2020).Google Scholar
- Wucherl Yoo, Shu Shi, Won J Jeon, Klara Nahrstedt, and Roy H Campbell. 2010. Real-time parallel remote rendering for mobile devices using graphics processing units. In 2010 IEEE International Conference on Multimedia and Expo. IEEE, 902--907.Google ScholarCross Ref
- Yizhong Zhang, Jiaolong Yang, Zhen Liu, Ruicheng Wang, Guojun Chen, Xin Tong, and Baining Guo. 2022. Virtualcube: An immersive 3d video communication system. IEEE Transactions on Visualization and Computer Graphics 28, 5 (2022), 2146--2156.Google ScholarCross Ref
- Zhengming Zhang, Yaoqing Yang, Meng Hua, Chunguo Li, Yongming Huang, and Luxi Yang. 2019. Proactive caching for vehicular multi-view 3D video streaming via deep reinforcement learning. IEEE Transactions on Wireless Communications 18, 5 (2019), 2693--2706.Google ScholarDigital Library
- Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, Federico Alvarez, and Petros Daras. 2019. Spherical view synthesis for self-supervised 360 depth estimation. In 2019 International Conference on 3D Vision (3DV). IEEE, 690--699.Google ScholarCross Ref
Index Terms
- ZGaming: Zero-Latency 3D Cloud Gaming by Image Prediction
Recommendations
A latency compensation technique based on game characteristics to mitigate the influence of delay on cloud gaming quality of experience
MMSys '20: Proceedings of the 11th ACM Multimedia Systems ConferenceCloud Gaming (CG) is an immersive multimedia service that promises many benefits. In CG, the games are rendered in a cloud server, and the resulted scenes are streamed as a video sequence to the client. Using CG users are not forced to update their ...
Subjective Evaluation of Latency and Packet Loss in a Cloud-Based Game
ITNG '13: Proceedings of the 2013 10th International Conference on Information Technology: New GenerationsOn-demand multimedia services are more popular than ever and continue to grow. Consumers can now stream music, movies, television, and video games at the push of a button. Such services typically require a minimum connection speed to support streaming. ...
Server Allocation for Multiplayer Cloud Gaming
MM '16: Proceedings of the 24th ACM international conference on MultimediaAdvances in cloud computing and GPU virtualization are allowing the game industry to move into a cloud gaming era. While shifting standalone video games to the cloud gaming mode is straightforward, adapting multiplayer online games to the cloud gaming ...
Comments