Skip to main content
SpringerLink
Log in

A hybrid graphics/video rate control method based on graphical assets for cloud gaming

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In Hybrid Cloud Gaming (HCG), as long as the graphical assets are available at the client side, rendering can be performed locally. However, if the client is not able to render all the frames in time, this may result in the frame rate to drop under the target value. This paper presents an Asset-based frame-level hybrid graphics/video rate control method for HCG, referred to as AHCG, which aims at improving the Quality of Experience (QoE) of players by tapping into the available processing power at the client side, while keeping a steady frame rate for thin clients and taking full advantage of the user’s tolerable delay. In the proposed client/server model, graphics data are intercepted and streamed in an asset-based approach. This approach handles the rate control issue per asset which is defined to be 3D object model data, textures, and shader programs. Rendering a frame on the client side not only maintains original quality for that frame, but it also reduces bandwidth requirements of the entire service by reusing the same assets for different frames. In the proposed method, if the accumulated rendering delay at the client side violates the tolerable delay set by the user, video streaming is used to compensate for the client device’s lack of processing power. Furthermore, quality fluctuation between graphics and video frames is addressed to provide a seamless experience when switching between graphics and video streaming. Several objective and subjective tests are conducted and the experimental results show a 20-fps increase in frame rate while maintaining a minimum value of 58 fps, with a minimum of 0.25 units improvement in the pooled standard deviation of SSIM values, compared to existing HCGs. Also, the subjective tests suggest an average 5.62 percent improvement in MOS compared to best HCG methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Huang, C.-Y., Chen, K.-T., Chen, D.-Y., Hsu, H.-J., Hsu, C.-H.: GamingAnywhere: the first open source cloud gaming system. ACM Trans. Multimed. Comput. Commun. Appl. 10(1s), 25 (2014). https://doi.org/10.1145/2537855

    Article  Google Scholar 

  2. Mishra, D., Zarki, M.E., Erbad, A., Hsu, C.-H., Venkatasubramanian, N.: Clouds + games: a multifaceted approach. IEEE Internet Comput. 18(3), 20–27 (2014). https://doi.org/10.1109/MIC.2014.20

    Article  Google Scholar 

  3. Shea, R., Liu, J., Ngai, E.C., Cui, Y.: Cloud gaming: architecture and performance. IEEE Netw. 27(4), 16–21 (2013). https://doi.org/10.1109/MNET.2013.6574660

    Article  Google Scholar 

  4. Ahmadi, H., Tootaghaj, S.Z., Hashemi, M.R., Shirmohammadi, S.: A game attention model for efficient bit rate allocation in cloud gaming. Multimedia Syst. 20(5), 485–501 (2014). https://doi.org/10.1007/s00530-014-0381-1

    Article  Google Scholar 

  5. Cuervo, E., Wolman, A., Cox, L.P., Lebeck, K., Razeen, A., Saroiu, S., Musuvathi, M.: Kahawai: high-quality mobile gaming using GPU offload. In: Proceedings of the 13th annual international conference on mobile systems, applications, and services, Florence, Italy 2015, pp. 121–135. Association for Computing Machinery (2015)

  6. Semsarzadeh, M., Yassine, A., Shirmohammadi, S.: Video encoding acceleration in cloud gaming. IEEE Trans. Circuits Syst. Video Technol. 25(12), 1975–1987 (2015). https://doi.org/10.1109/TCSVT.2015.2452778

    Article  Google Scholar 

  7. Nave, I., David, H., Shani, A., Tzruya, Y., Laikari, A., Eisert, P., Fechteler, P.: Games@large graphics streaming architecture. In: 2008 IEEE International Symposium on Consumer Electronics, 14–16 April 2008, pp. 1–4 (2008)

  8. Barman, N., Martini, M.G., Zadtootaghaj, S., Möller, S., Lee, S.: A comparative quality assessment study for gaming and non-gaming videos. In: 2018 Tenth international conference on quality of multimedia experience (QoMEX), Cagliari, Italy, 29 May–1 June 2018 2018, pp. 1–6. IEEE (2018)

  9. Zadtootaghaj, S., Schmidt, S., Möller, S.: Modeling gaming QoE: towards the impact of frame rate and bit rate on cloud gaming. In: 2018 Tenth international conference on quality of multimedia experience (QoMEX), 29 May-1 June 2018, pp. 1–6 (2018)

  10. Chan, K.L., Ichikawa, K., Watashiba, Y., Putchong, U., Iida, H.: A hybrid-streaming method for cloud gaming: to improve the graphics quality delivered on highly accessible game contents. Int. J. Serious Games (2017). https://doi.org/10.17083/ijsg.v4i2.163

    Article  Google Scholar 

  11. Nan, X., Guo, X., Lu, Y., He, Y., Guan, L., Li, S., Guo, B.: Delay–rate–distortion optimization for cloud gaming with hybrid streaming. IEEE Trans. Circuits Syst. Video Technol. 27(12), 2687–2701 (2016). https://doi.org/10.1109/TCSVT.2016.2595330

    Article  Google Scholar 

  12. Nan, X., Guo, X., Lu, Y., He, Y., Guan, L., Li, S., Guo, B.: A novel cloud gaming framework using joint video and graphics streaming. In: 2014 IEEE international conference on multimedia and expo (ICME), 14–18 July 2014, pp. 1–6 (2014)

  13. Mohammadi, I.S., Ghanbari, M., Hashemi, M.R.: An API-level frame workload model for real-time rendering applications. https://doi.org/10.36227/techrxiv.12278666.v1 (2020)

  14. Sun, L., Osman, H.A., Lang, J.: A hybrid remote rendering method for mobile applications. Multimedia Tools Appl. 79(5), 3333–3358 (2019). https://doi.org/10.1007/s11042-019-7306-0

    Article  Google Scholar 

  15. Tan, K.T., Ghanbari, M., Pearson, D.E.: An objective measurement tool for MPEG video quality. Signal Process. 70(3), 279–294 (1998). https://doi.org/10.1016/S0165-1684(98)00129-7

    Article  MATH  Google Scholar 

  16. Union, I.T.: Subjective evaluation methods for gaming quality. In: Telecommunication Standardization Sector of ITU, p. 34 (2018)

  17. Liao, X., Lin, L., Tan, G., Jin, H., Yang, X., Zhang, W., Li, B.: LiveRender: a cloud gaming system based on compressed graphics streaming. IEEE/ACM Trans. Netw. 24(4), 2128–2139 (2016). https://doi.org/10.1109/TNET.2015.2450254

    Article  Google Scholar 

  18. Chen, D.-Y., El-Zarki, M.: A framework for adaptive residual streaming for single-player cloud gaming. ACM Trans. Multimed. Comput. Commun. Appl. 15(2s), 23 (2019). https://doi.org/10.1145/3336498

    Article  Google Scholar 

  19. Illahi, G., Siekkinen, M., Masala, E.: Foveated video streaming for cloud gaming. In: 2017 IEEE 19th international workshop on multimedia signal processing (MMSP), 16–18 Oct 2017, pp. 1–6 (2017)

  20. Ke, Y., Ye, G., Wu, D., Zhou, Y., Ngai, E., Hu, H.: GECKO: gamer experience-centric bitrate control algorithm for cloud gaming. In: Image and graphics, pp. 325–335. Springer, Cham (2017)

    Chapter  Google Scholar 

  21. Wang, P., Ellis, A.I., Hart, J.C., Hsu, C.: Optimizing next-generation cloud gaming platforms with planar map streaming and distributed rendering. In: 2017 15th Annual workshop on network and systems support for games (NetGames), 22–23 June 2017, pp. 1–6 (2017)

  22. Hegazy, M., Diab, K., Saeedi, M., Ivanovic, B., Amer, I., Liu, Y., Sines, G., Hefeeda, M.: Content-aware video encoding for cloud gaming. In: Paper presented at the proceedings of the 10th ACM multimedia systems conference, Amherst, Massachusetts

  23. Illahi, G.K., Gemert, T.V., Siekkinen, M., Masala, E., Oulasvirta, A., Ylä-Jääski, A.: Cloud gaming with foveated video encoding. ACM Trans. Multimedia Comput. Commun. Appl. (2020). https://doi.org/10.1145/3369110

    Article  Google Scholar 

  24. Nguyen, D.V., Tran, H.T.T., Thang, T.C.: A delay-aware adaptation framework for cloud gaming under the computation constraint of user devices. In: MultiMedia modeling, pp. 27–38. Springer, Cham (2020)

    Chapter  Google Scholar 

  25. Molnar, S., Cox, M., Ellsworth, D., Fuchs, H.: A sorting classification of parallel rendering. IEEE Comput. Graph. Appl. 14(4), 23–32 (1994). https://doi.org/10.1109/38.291528

    Article  Google Scholar 

  26. Akenine-Möller, T., Haines, E., Hoffman, N.: Real-time rendering, 4th edn. CRC Press (2018)

    Book  Google Scholar 

  27. SLI. https://docs.nvidia.com/gameworks/content/technologies/desktop/sli.htm (2004). Accessed Feb 2020

  28. CrossFire Technology. https://www.amd.com/en/technologies/crossfire (2005). Accessed February 2020

  29. Hoppe, H.: Progressive Meshes. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques 1996, pp. 99–108. Association for Computing Machinery

  30. Sullivan, G.J., Wiegand, T.: Rate-distortion optimization for video compression. IEEE Signal Process. Mag. 15(6), 74–90 (1998). https://doi.org/10.1109/79.733497

    Article  Google Scholar 

  31. Ghandi, M.M., Ghanbari, M.: A Lagrangian optimized rate control algorithm for the H.264/AVC encoder. In: 2004 International conference on image processing, 2004. ICIP '04., Singapore, Singapore October 2004, pp. 123–126. IEEE

  32. Schmidt, S., Zadtootaghaj, S., Möller, S.: Towards the delay sensitivity of games: there is more than genres. In: 2017 Ninth international conference on quality of multimedia experience (QoMEX), 31 May–2 June 2017, pp. 1–6 (2017)

  33. Lee, H.-J., Chiang, T., Zhang, Y.-Q.: Scalable rate control for MPEG-4 video. IEEE Trans. Circuits Syst. Video Technol. 10(6), 878–894 (2000). https://doi.org/10.1109/76.867926

    Article  Google Scholar 

  34. Ma, S., Wen, G., Yan, L.: Rate-distortion analysis for H.264/AVC video coding and its application to rate control. IEEE Trans. Circuits Syst. Video Technol. 15(12), 1533–1544 (2005). https://doi.org/10.1109/TCSVT.2005.857300

    Article  Google Scholar 

  35. Sabet, S.S., Schmidt, S., Zadtootaghaj, S., Griwodz, C., Moller, S.: Towards applying game adaptation to decrease the impact of delay on quality of experience. In: 2018 IEEE international symposium on multimedia (ISM), 10–12 Dec 2018, pp. 114–121 (2018)

  36. Sabet, S.S., Schmidt, S., Griwodz, C., Möller, S.: Towards the impact of Gamers' adaptation to delay variation on gaming quality of experience. In: 2019 Eleventh international conference on quality of multimedia experience (QoMEX), 5–7 June 2019, pp. 1–6 (2019)

  37. Li, C., Wu, D., Xiong, H.: Delay–power-rate-distortion model for wireless video communication under delay and energy constraints. IEEE Trans. Circuits Syst. Video Technol. 24(7), 1170–1183 (2014). https://doi.org/10.1109/TCSVT.2014.2302517

    Article  Google Scholar 

  38. Boyd, S., Vandenberghe, L.: Convex optimization. Cambridge University Press, Cambridge (2004)

    Book  Google Scholar 

  39. Eu, Y.X., Tanu, J., Law, J.J., Ghazali, M.H.B., Tay, S.S., Ooi, W.T., Bhojan, A.: SuperStreamer: enabling progressive content streaming in a game engine. Paper presented at the proceedings of the 24th ACM international conference on Multimedia, Amsterdam, The Netherlands

  40. APITrace. https://github.com/apitrace (2019). Accessed 19 Apr 2020

  41. DirectX Graphics and Gaming | Microsoft Docs. https://docs.microsoft.com/en-us/windows/desktop/directx (2018). Accessed 24 Feb 2020

  42. team, x.: x264. https://www.videolan.org/developers/x264.html (2004). Accessed 11 Aug 2019

  43. Zhou, W., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  44. Zadtootaghaj, S., Schmidt, S., Barman, N., Möller, S., Martini, M.G.: A classification of video games based on game characteristics linked to video coding complexity. In: 2018 16th Annual workshop on network and systems support for games (NetGames), 12–15 June 2018, pp. 1–6 (2018)

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran

    Iman Soltani Mohammadi, Mohammad Ghanbari & Mahmoud Reza Hashemi

  2. School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK

    Mohammad Ghanbari

Authors
  1. Iman Soltani Mohammadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Ghanbari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mahmoud Reza Hashemi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mahmoud Reza Hashemi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohammadi, I.S., Ghanbari, M. & Hashemi, M.R. A hybrid graphics/video rate control method based on graphical assets for cloud gaming. J Real-Time Image Proc 19, 41–59 (2022). https://doi.org/10.1007/s11554-021-01159-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01159-y

Keywords

Search

Navigation