Skip to main content
Springer Nature Link
Log in

Identifying gameplay videos that exhibit bugs in computer games

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

With the rapid growing market and competition in the gaming industry, it is challenging to develop a successful game, making the quality of games very important. To improve the quality of games, developers commonly use gamer-submitted bug reports to locate bugs in games. Recently, gameplay videos have become popular in the gaming community. A few of these videos showcase a bug, offering developers a new opportunity to collect context-rich bug information. In this paper, we investigate whether videos that showcase a bug can automatically be identified from the metadata of gameplay videos that are readily available online. Such bug videos could then be used as a supplemental source of bug information for game developers. We studied the number of gameplay videos on the Steam platform, one of the most popular digital game distribution platforms, and the difficulty of identifying bug videos from these gameplay videos. We show that naïve approaches such as using keywords to search for bug videos are time-consuming and imprecise. We propose an approach which uses a random forest classifier to rank gameplay videos based on their likelihood of being a bug video. Our proposed approach achieves a precision that is 43% higher than that of the naïve keyword searching approach on a manually labelled dataset of 96 videos. In addition, by evaluating 1,400 videos that are identified by our approach as bug videos, we calculated that our approach has both a mean average precision at 10 and a mean average precision at 100 of 0.91. Our study demonstrates that it is feasible to automatically identify gameplay videos that showcase a bug.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. https://gaming.youtube.com/

  2. Games with the highest budgets for development and promotion.

  3. https://www.youtube.com/watch?v=W9in3AEYuJk

  4. https://www.youtube.com/watch?v=pnFuzq8Ii8o

  5. http://steamcommunity.com/id/GamesGlitches/

  6. https://www.twitch.tv/

References

  • Afic R (2015) How to report a game bug. https://boards.na.leagueoflegends.com/en/c/bug-report/3mQGBEjA-how-to-report-a-game-bug, (last visited: Mar 25 2019)

  • Alexa (2018) Youtube.com traffic, demographics and competitors - alexa. https://www.alexa.com/siteinfo/youtube.com, (last visited: Mar 25 2019)

  • Baeza-Yates R, Ribeiro-Neto B et al (1999) Modern information retrieval, vol 463. ACM Press, New York

    Google Scholar 

  • Blackburn J, Kourtellis N, Skvoretz J, Ripeanu M, Iamnitchi A (2014) Cheating in online games: a social network perspective. ACM Trans Internet Technol (TOIT) 13(3):9

    Article  Google Scholar 

  • Habakuk CCP (2017) Bug reporting - eve community. https://community.eveonline.com/support/test-servers/bug-reporting/, (last visited: Mar 25 2019)

  • Chambers C, Feng Wc, Sahu S, Saha D (2005) Measurement-based characterization of a collection of on-line games. In: Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement, USENIX Association, pp 1–14

  • Cleveland W S, Devlin S J (1988) Locally weighted regression: an approach to regression analy sis by local fitting. J Amer Stat Assoc 83(403):596–610

    Article  Google Scholar 

  • Efron B (1983) Estimating the error rate of a prediction rule: improvement on cross-validation. J Amer Stat Assoc 78(382):316–331

    Article  MathSciNet  Google Scholar 

  • Ekaterina P, Gross N (2017) 4 reasons people watch gaming content on Youtube. https://www.thinkwithgoogle.com/consumer-insights/statistics-youtube-gaming-content/ (last visited: Mar 25 2019)

  • Electronic Arts Inc (2018) Origin. https://www.origin.com, (last visited: Mar 25 2019)

  • Fulda N, Ricks D, Murdoch B, Wingate D (2018) Threat, explore, barter, puzzle: A semantically-informed algorithm for extracting interaction modes. In: The Workshops of the 32nd AAAI Conference on Artificial Intelligence, pp 1–5

  • Google (2018) Irrelevant keywords - search console help. https://support.google.com/webmasters/answer/66358, (last visited: Mar 25 2019)

  • Guzdial M, Riedl M (2016) Game level generation from gameplay videos. In: Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference

  • Guzdial M, Li B, Riedl MO (2017) Game engine learning from video. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI), pp 3707–3713

  • Guzdial M, Shah S, Riedl M (2018) Towards automated let’s play commentary. CoRR arXiv:1809.09424

  • Hilvert-Bruce Z, Neill JT, Sjöblom M, Hamari J (2018) Social motivations of live-streaming viewer engagement on Twitch. Comput Hum Behav 84:58–67

    Article  Google Scholar 

  • Ho TK (1995) Random decision forests. In: Proceedings of the 3rd International Conference on Document Analysis and Recognition. IEEE, vol 1, pp 278–282

  • Hooimeijer P, Weimer W (2007) Modeling bug report quality. In: Proceedings of the 22nd IEEE/ACM international conference on Automated software engineering, ACM, pp 34–43

  • Jacob LB, Kohwalter TC, Machado AFV, Clua EWG, d Oliveira D (2014) A non-intrusive approach for 2d platform game design analysis based on provenance data extracted from game streaming. In: Brazilian Symposium on Computer Games and Digital Entertainment, pp 41–50

  • Kabinna S, Bezemer CP, Shang W, Syer MD, Hassan AE (2018) Examining the stability of logging statements. Empir Softw Eng 23(1):290–333

    Article  Google Scholar 

  • Kempka M, Wydmuch M, Runc G, Toczek J, Jaśkowski W (2016) ViZDoom: A Doom-based AI research platform for visual reinforcement learning. In: IEEE Conference on Computational Intelligence and Games (CIG), pp 1–8

  • Larson R R (2010) Introduction to information retrieval. J Amer Soc Inf Sci Technol 61(4):852–853

    Google Scholar 

  • Lewis C, Whitehead J, Wardrip-Fruin N (2010) What went wrong: a taxonomy of video game bugs. In: Proceedings of the 5th International Conference on the Foundations of Digital Games, ACM, pp 108–115

  • Lin D, Bezemer CP, Hassan AE (2017) Studying the urgent updates of popular games on the Steam platform. Empir Softw Eng 22(4):2095–2126

    Article  Google Scholar 

  • Lin D, Bezemer CP, Hassan AE (2018) An empirical study of early access games on the Steam platform. Empir Softw Eng 23(2):771–799

    Article  Google Scholar 

  • Lin D, Bezemer CP, Hassan AE (2019a) Supplementary material for our paper. https://github.com/SAILResearch/suppmaterial-19-dayi-game_video, (last visited: Mar 25 2019)

  • Lin D, Bezemer CP, Zou Y, Hassan AE (2019b) An empirical study of game reviews on the Steam platform. Empir Softw Eng 24(1):170–207

    Article  Google Scholar 

  • Linstead E, Baldi P (2009) Mining the coherence of GNOME bug reports with statistical topic models. In: 6th IEEE International Working Conference on Mining Software Repositories. IEEE, pp 99–102

  • Long JD, Feng D, Cliff N (2003) Ordinal Analysis of Behavioral Data. Wiley, New York

    Book  Google Scholar 

  • Louppe G, Wehenkel L, Sutera A, Geurts P (2013) Understanding variable importances in forests of randomized trees. In: Advances in Neural Information Processing Systems, pp 431–439

  • Luo Z, Guzdial M, Liao N, Riedl M (2018) Player experience extraction from gameplay video. In: AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pp 1–7

  • Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M A (2013) Playing Atari with deep reinforcement learning. CoRR arXiv:1312.5602

  • NLTK Project (2017) Natural Language Toolkit. https://www.nltk.org/, (last visited: Mar 25, 2019)

  • Phillips T (2018) The human cost of Red Dead Redemption, 2. https://www.eurogamer.net/articles/2018-10-25-the-human-cost-of-red-dead-redemption-2, (last visited: Mar 25 2019)

  • PwC (2016) Value of the global video games market from 2011 to 2020 (in billion u.s. dollars). https://www.statista.com/statistics/246888/value-of-the-global-video-game-market/, (last visited: Mar 25 2019)

  • Romano J, Kromrey J D, Coraggio J, Skowronek J, Devine L (2006) Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’s d indices the most appropriate choices. In: Annual meeting of the Southern Association for Institutional Research

  • Schreier J (2018) Inside Rockstar Games,’ culture of crunch. https://kotaku.com/inside-rockstar-games-culture-of-crunch-1829936466, (last visited: Mar 25 2019)

  • Sixen (2010) How to write a good bug report. https://us.battle.net/forums/en/sc2/topic/188789092, (last visited: Mar 25 2019)

  • Sjöblom M, Hamari J (2017) Why do people watch others play video games? An empirical study on the motivations of Twitch users. Comput Hum Behav 75:985–996

    Article  Google Scholar 

  • Summerville A, Guzdial M, Mateas M, Riedl MO (2016) Learning player tailored content from observation: Platformer level generation from video traces using lstms. In: 12th Artificial Intelligence and Interactive Digital Entertainment Conference

  • Tantithamthavorn C, McIntosh S, Hassan A E, Matsumoto K (2017) An empirical comparison of model validation techniques for defect prediction models. IEEE Trans Softw Eng 43(1):1–18

    Article  Google Scholar 

  • Tantithamthavorn C, Hassan AE (2018) An experience report on defect modelling in practice: Pitfalls and challenges. In: Inproceedings of the International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP’18). ACM, pp 286–295

  • Valve (2018a) Steam community. https://steamcommunity.com/, (last visited: Mar 25, 2019)

  • Valve (2018b) Steam store. https://store.steampowered.com/, (last visited: Mar 25, 2019)

  • Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bullet 1(6):80–83

    Article  Google Scholar 

  • YouTube (2018) Auto-generated topic channels - youtube help. https://support.google.com/youtube/answer/2579942, (last visited: Mar 25 2019)

  • Zimmermann T, Premraj R, Sillito J, Breu S (2009) Improving bug tracking systems. In: 31st International Conference on Software Engineering-Companion Volume. IEEE, pp 247–250

  • Zimmermann T, Premraj R, Bettenburg N, Just S, Schroter A, Weiss C (2010) What makes a good bug report? IEEE Trans Softw Eng 36(5):618–643

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Software Analysis and Intelligence Lab (SAIL), Queen’s University, Kingston, ON, Canada

    Dayi Lin & Ahmed E. Hassan

  2. Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB, Canada

    Cor-Paul Bezemer

Authors
  1. Dayi Lin
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Cor-Paul Bezemer
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Ahmed E. Hassan
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Dayi Lin.

Additional information

Communicated by: Emerson Murphy-Hill

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, D., Bezemer, CP. & Hassan, A.E. Identifying gameplay videos that exhibit bugs in computer games. Empir Software Eng 24, 4006–4033 (2019). https://doi.org/10.1007/s10664-019-09733-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-019-09733-6

Keywords