skip to main content
research-article

Post-Render Warp with Late Input Sampling Improves Aiming Under High Latency Conditions

Published: 26 August 2020 Publication History

Abstract

End-to-end latency in remote-rendering systems can reduce user task performance. This notably includes aiming tasks on game streaming services, which are presently below the standards of competitive first-person desktop gaming. We evaluate the latency-induced penalty on task completion time in a controlled environment and show that it can be significantly mitigated by adopting and modifying image and simulation-warping techniques from virtual reality, eliminating up to 80% of the penalty from 80 ms of added latency. This has potential to enable remote rendering for esports and increase the effectiveness of remote-rendered content creation and robotic teleoperation. We provide full experimental methodology, analysis, implementation details, and source code.

Supplementary Material

kim (kim.zip)
Supplemental movie, appendix, image and software files for, Post-Render Warp with Late Input Sampling Improves Aiming Under High Latency Conditions
MP4 File (3406187.mp4)
Presentation Video

References

[1]
Sudhir Aggarwal, Hemant Banavar, Amit Khandelwal, Sarit Mukherjee, and Sampath Rangarajan. 2004. Accuracy in Dead-Reckoning Based Distributed Multi-Player Games. In Proceedings of 3rd ACM SIGCOMM Workshop on Network and System Support for Games (Portland, Oregon, USA) (NetGames '04). Association for Computing Machinery, New York, NY, USA, 161--165. https://doi.org/10.1145/1016540.1016559
[2]
Dmitry Andreev. 2010. Real-Time Frame Rate up-Conversion for Video Games: Or How to Get from 30 to 60 Fps for "Free". In ACM SIGGRAPH 2010 Talks (Los Angeles, California) (SIGGRAPH '10). Association for Computing Machinery, New York, NY, USA, Article 16, 1 pages. https://doi.org/10.1145/1837026.1837047
[3]
Dean Beeler, Ed Hutchins, and Paul Pedriana. 2016. Asynchronous Spacewarp. https://developer.oculus.com/blog/asynchronous-spacewarp/ Oculus Blog Post.
[4]
Tom Beigbeder, Rory Coughlan, Corey Lusher, John Plunkett, Emmanuel Agu, and Mark Claypool. 2004. The Effects of Loss and Latency on User Performance in Unreal Tournament 2003®. In Proceedings of 3rd ACM SIGCOMM Workshop on Network and System Support for Games (Portland, Oregon, USA) (NetGames '04). Association for Computing Machinery, New York, NY, USA, 144--151. https://doi.org/10.1145/1016540.1016556
[5]
Yahn W. Bernier. 2003. Latency Compensating Methods in Client/Server In-game Protocol Design and Optimization. (2003).
[6]
Kuan-Ta Chen, Polly Huang, and Chin-Laung Lei. 2006. How sensitive are online gamers to network quality? Commun. ACM 49 (11 2006), 34--38. https://doi.org/10.1145/1167838.1167859
[7]
Shenchang Eric Chen and Lance Williams. 1993. View Interpolation for Image Synthesis. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Anaheim, CA) (SIGGRAPH '93). Association for Computing Machinery, New York, NY, USA, 279--288. https://doi.org/10.1145/166117.166153
[8]
Sharon Choy, Bernard Wong, Gwendal Simon, and Catherine Rosenberg. 2012. The Brewing Storm in Cloud Gaming: A Measurement Study on Cloud to End-User Latency. In Proceedings of the 11th Annual Workshop on Network and Systems Support for Games (Venice, Italy) (NetGames '12). IEEE Press, Article 2, 6 pages.
[9]
Sharon Choy, Bernard Wong, Gwendal Simon, and Catherine Rosenberg. 2014. A Hybrid Edge-Cloud Architecture for Reducing On-Demand Gaming Latency. Multimedia Systems 20 (10 2014), 503--519. https://doi.org/10.1007/s00530-014-0367-z
[10]
Mark Claypool. 2005. The Effect of Latency on User Performance in Real-Time Strategy Games. Comput. Netw. 49, 1 (Sept. 2005), 52--70.
[11]
Mark Claypool and Kajal Claypool. 2006. Latency and Player Actions in Online Games. Commun. ACM 49, 11 (Nov. 2006), 40--45. https://doi.org/10.1145/1167838.1167860
[12]
Mark Claypool and Kajal Claypool. 2010. Latency Can Kill: Precision and Deadline in Online Games. In Proceedings of the First Annual ACM SIGMM Conference on Multimedia Systems (Phoenix, Arizona, USA) (MMSys '10). Association for Computing Machinery, New York, NY, USA, 215--222. https://doi.org/10.1145/1730836.1730863
[13]
Piotr Didyk, Elmar Eisemann, Tobias Ritschel, Karol Myszkowski, and Hans-Peter Seidel. 2010. Perceptually-motivated realtime temporal upsampling of 3D content for high-refresh-rate displays. Computer Graphics Forum (Proc. of Eurographics) 29, 2 (2010), 713--722. http://graphics.tudelft.nl/Publications-new/2010/DERMS10c
[14]
Sebastian Friston, Tobias Ritschel, and Anthony Steed. 2019. Perceptual Rasterization for Head-Mounted Display Image Synthesis. ACM Trans. Graph. 38, 4, Article 97 (July 2019), 14 pages. https://doi.org/10.1145/3306346.3323033
[15]
Benjamin Goyette. 2016. Fighting Latency on Call of Duty Black Ops III. (2016). https://www.gdcvault.com/play/1023220/GDC.
[16]
Akimitsu Hogge. 2019. Controller to Display Latency in 'Call of Duty'. (2019). https://www.gdcvault.com/play/1026327/GDC.
[17]
Zenja Ivkovic, Ian Stavness, Carl Gutwin, and Steven Sutcliffe. 2015. Quantifying and Mitigating the Negative Effects of Local Latencies on Aiming in 3D Shooter Games. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 135--144. https://doi.org/10.1145/2702123.2702432
[18]
Teemu Kämäräinen, Matti Siekkinen, Jukka Eerikäinen, and Antti Ylä-Jääski. 2018. CloudVR: Cloud Accelerated Interactive Mobile Virtual Reality. In Proceedings of the 26th ACM International Conference on Multimedia (Seoul, Republic of Korea) (MM '18). Association for Computing Machinery, New York, NY, USA, 1181--1189. https://doi.org/10.1145/3240508.3240620
[19]
Babis Koniaris, Maggie Kosek, David Sinclair, and Kenny Mitchell. 2017. Real-Time Rendering with Compressed Animated Light Fields. In Proceedings of the 43rd Graphics Interface Conference (Edmonton, Alberta, Canada) (GI '17). Canadian Human-Computer Communications Society, Waterloo, CAN, 33--40.
[20]
Z. Lai, Y. C. Hu, Y. Cui, L. Sun, N. Dai, and H. Lee. 2019. Furion: Engineering High-Quality Immersive Virtual Reality on Today's Mobile Devices. IEEE Transactions on Mobile Computing (2019), 1--1.
[21]
John R. Lange, Peter A. Dinda, and Samuel Rossoff. 2008. Experiences with Client-Based Speculative Remote Display. In USENIX 2008 Annual Technical Conference (Boston, Massachusetts) (ATC'08). USENIX Association, USA, 419--432.
[22]
Kyungmin Lee, David Chu, Eduardo Cuervo, Johannes Kopf, Yury Degtyarev, Sergey Grizan, Alec Wolman, and Jason Flinn. 2015. Outatime: Using Speculation to Enable Low-Latency Continuous Interaction for Mobile Cloud Gaming. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (Florence, Italy) (MobiSys '15). Association for Computing Machinery, New York, NY, USA, 151--165. https://doi.org/10.1145/2742647.2742656
[23]
Andrew Lippman. 1980. Movie-Maps: An Application of the Optical Videodisc to Computer Graphics. SIGGRAPH Comput. Graph. 14, 3 (July 1980), 32--42. https://doi.org/10.1145/965105.807465
[24]
Gerrit Lochmann, Bernhard Reinert, Tobias Ritschel, Stefan Müller, and Hans-Peter Seidel. 2014. Real-time Reflective and Refractive Novel-view Synthesis. In VMV. 9--16.
[25]
Michael Long and Carl Gutwin. 2018. Characterizing and Modeling the Effects of Local Latency on Game Performance and Experience. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (Melbourne, VIC, Australia) (CHI PLAY '18). Association for Computing Machinery, New York, NY, USA, 285--297. https://doi.org/10.1145/3242671.3242678
[26]
I Scott MacKenzie. 2012. Human-computer interaction: An empirical research perspective. Newnes.
[27]
William R. Mark, Leonard McMillan, and Gary Bishop. 1997. Post-Rendering 3D Warping. In Proceedings of the 1997 Symposium on Interactive 3D Graphics (Providence, Rhode Island, USA) (I3D '97). Association for Computing Machinery, New York, NY, USA, 7-ff. https://doi.org/10.1145/253284.253292
[28]
Joanna Nelius. 2020. PCGamer GeForce Now beats Statdia in our input latency testing. https:https://www.pcgamer.com/geforce-now-beats-stadia-in-our-input-latency-testing/. Accessed: 2020-04-13.
[29]
Newzoo. 2020. Global esports market report. Vol. April 2020 Update. Newzoo.
[30]
Oculus Blog, Facebook 2019. Introducing ASW 2.0: Better Accuracy, Lower Latency. Oculus Blog, Facebook. https://www.oculus.com/blog/introducing-asw-2-point-0-better-accuracy-lower-latency/?fb_comment_id=2229988523715492_2230021217045556.
[31]
Kjetil Raaen and Andreas Petlund. 2015. How Much Delay is There Really in Current Games?. In Proceedings of the 6th ACM Multimedia Systems Conference (Portland, Oregon) (MMSys '15). Association for Computing Machinery, New York, NY, USA, 89--92. https://doi.org/10.1145/2713168.2713188
[32]
Bernhard Reinert, Johannes Kopf, Tobias Ritschel, Eduardo Cuervo, David Chu, and Hans-Peter Seidel. 2016. Proxy-guided image-based rendering for mobile devices. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 353--362.
[33]
Riot Games. 2020. Valorant.
[34]
Cheryl Savery and T. C. Graham. 2013. Timelines: Simplifying the Programming of Lag Compensation for the next Generation of Networked Games. Multimedia Syst. 19, 3 (June 2013), 271--287. https://doi.org/10.1007/s00530-012-0271-3
[35]
Andre Schollmeyer, Simon Schneegans, Stephan Beck, Anthony Steed, and Bernd Froehlich. 2017. Efficient Hybrid Image Warping for High Frame-Rate Stereoscopic. IEEE Transactions on Visualization and Computer Graphics PP (01 2017), 1--1. https://doi.org/10.1109/TVCG.2017.2657078
[36]
Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. 1998. Layered Depth Images. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '98). Association for Computing Machinery, New York, NY, USA, 231--242. https://doi.org/10.1145/280814.280882
[37]
Nathan Sheldon, Eric Girard, Seth Borg, Mark Claypool, and Emmanuel Agu. 2003. The Effect of Latency on User Performance in Warcraft III. In Proceedings of the 2nd Workshop on Network and System Support for Games (Redwood City, California) (NetGames '03). Association for Computing Machinery, New York, NY, USA, 3--14. https://doi.org/10.1145/963900.963901
[38]
Josef Spjut, Ben Boudaoud, Kamran Binaee, Jonghyun Kim, Alexander Majercik, Morgan McGuire, David Luebke, and Joohwan Kim. 2019a. Latency of 30 ms Benefits First Person Targeting Tasks More Than Refresh Rate Above 60 Hz. In SIGGRAPH Asia 2019 Technical Briefs (Brisbane, QLD, Australia) (SA '19). Association for Computing Machinery, New York, NY, USA, 110--113. https://doi.org/10.1145/3355088.3365170
[39]
Josef Spjut, Ben Boudaoud, Kamran Binaee, Alexander Majercik, Morgan McGuire, and Joohwan Kim. 2019b. FirstPerson-Science: Quantifying Psychophysics for First Person Shooter Tasks. In UCI Esports Conference.
[40]
Josef Spjut, Ben Boudaoud, Pyarelal Knowles, Zander Majercik, Morgan McGuire, and Joohwan Kim. 2019c. FirstPerson-Science. https://github.com/NVlabs/abstract-fps
[41]
Art Tevs, Ivo Ihrke, and Hans-Peter Seidel. 2008. Maximum Mipmaps for Fast, Accurate, and Scalable Dynamic Height Field Rendering. In Proceedings of the 2008 Symposium on Interactive 3D Graphics and Games (Redwood City, California) (I3D '08). Association for Computing Machinery, New York, NY, USA, 183--190. https://doi.org/10.1145/1342250.1342279
[42]
Valve Software. 2012. Counter-Strike: Global Offensive.
[43]
J. M. P. van Waveren. 2016. The Asynchronous Time Warp for Virtual Reality on Consumer Hardware. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (Munich, Germany) (VRST '16). Association for Computing Machinery, New York, NY, USA, 37--46. https://doi.org/10.1145/2993369.2993375

Cited By

View all
  • (2024)GFFE: G-buffer Free Frame Extrapolation for Low-latency Real-time RenderingACM Transactions on Graphics10.1145/368792343:6(1-15)Online publication date: 19-Nov-2024
  • (2024)Exploring Patient-Generated Annotations to Digital Clinical Symptom Measures for Patient-Centered CommunicationProceedings of the ACM on Human-Computer Interaction10.1145/36869978:CSCW2(1-26)Online publication date: 8-Nov-2024
  • (2024)MOJI: Enhancing Emoji Search System with Query Expansions and Emoji RecommendationsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650838(1-8)Online publication date: 11-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Computer Graphics and Interactive Techniques
Proceedings of the ACM on Computer Graphics and Interactive Techniques  Volume 3, Issue 2
August 2020
218 pages
EISSN:2577-6193
DOI:10.1145/3420254
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 August 2020
Published in PACMCGIT Volume 3, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. esports
  2. latency
  3. streaming

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)56
  • Downloads (Last 6 weeks)18
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)GFFE: G-buffer Free Frame Extrapolation for Low-latency Real-time RenderingACM Transactions on Graphics10.1145/368792343:6(1-15)Online publication date: 19-Nov-2024
  • (2024)Exploring Patient-Generated Annotations to Digital Clinical Symptom Measures for Patient-Centered CommunicationProceedings of the ACM on Human-Computer Interaction10.1145/36869978:CSCW2(1-26)Online publication date: 8-Nov-2024
  • (2024)MOJI: Enhancing Emoji Search System with Query Expansions and Emoji RecommendationsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650838(1-8)Online publication date: 11-May-2024
  • (2024)CloVR: Fast-Startup Low-Latency Cloud VRIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337205930:5(2337-2346)Online publication date: 4-Mar-2024
  • (2024)Networked Metaverse Systems: Foundations, Gaps, Research DirectionsIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.34260985(5488-5539)Online publication date: 2024
  • (2023)PEDM: A Multi-task Learning Model for Persona-aware Emoji-embedded Dialogue GenerationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/357181919:3s(1-21)Online publication date: 24-Feb-2023
  • (2023)Literature Review: Latency Compensation Techniques for Online Gaming under an IoT Approach2023 IEEE Colombian Caribbean Conference (C3)10.1109/C358072.2023.10436174(1-6)Online publication date: 22-Nov-2023
  • (2022)Display Size and Targeting Performance: Small Hurts, Large May HelpSIGGRAPH Asia 2022 Conference Papers10.1145/3550469.3555396(1-8)Online publication date: 29-Nov-2022
  • (2022)FirstPersonScience: An Open Source Tool for Studying FPS Esports AimingACM SIGGRAPH 2022 Talks10.1145/3532836.3536233(1-2)Online publication date: 27-Jul-2022
  • (2022)A Survey and Taxonomy of Latency Compensation Techniques for Network Computer GamesACM Computing Surveys10.1145/351902354:11s(1-34)Online publication date: 9-Sep-2022
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media