Skip to main content
SpringerLink
Log in

An empirical study of game reviews on the Steam platform

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

The steadily increasing popularity of computer games has led to the rise of a multi-billion dollar industry. Due to the scale of the computer game industry, developing a successful game is challenging. In addition, prior studies show that gamers are extremely hard to please, making the quality of games an important issue. Most online game stores allow users to review a game that they bought. Such reviews can make or break a game, as other potential buyers often base their purchasing decisions on the reviews of a game. Hence, studying game reviews can help game developers better understand user concerns, and further improve the user-perceived quality of games. In this paper, we perform an empirical study of the reviews of 6224 games on the Steam platform, one of the most popular digital game delivery platforms, to better understand if game reviews share similar characteristics with mobile app reviews, and thereby understand whether the conclusions and tools from mobile app review studies can be leveraged by game developers. In addition, new insights from game reviews could possibly open up new research directions for research of mobile app reviews. We first conduct a preliminary study to understand the number of game reviews and the complexity to read through them. In addition, we study the relation between several game-specific characteristics and the fluctuations of the number of reviews that are received on a daily basis. We then focus on the useful information that can be acquired from reviews by studying the major concerns that users express in their reviews, and the amount of play time before players post a review. We find that game reviews are different from mobile app reviews along several aspects. Additionally, the number of playing hours before posting a review is a unique and helpful attribute for developers that is not found in mobile app reviews. Future longitudinal studies should be conducted to help developers and researchers leverage this information. Although negative reviews contain more valuable information about the negative aspects of the game, such as mentioned complaints and bug reports, developers and researchers should also not ignore the potentially useful information in positive reviews. Our study on game reviews serves as a starting point for other game review researchers, and suggests that prior studies on mobile app reviews may need to be revisited.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Listing 1
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. http://store.steampowered.com/app/413150

  2. http://store.steampowered.com/app/730

  3. http://store.steampowered.com/app/371310

  4. http://store.steampowered.com/app/438340

  5. http://store.steampowered.com/earlyaccessfaq/

  6. Anyone who purchased the game is the owner of the game, but only the people who played the game on that day are counted as the player of the game.

  7. https://www.nintendo.com/games/

References

  • Alden (2017) Steamworks Partner Program. https://partner.steamgames.com/steamdirect, (last visited: Mar 29, 2018)

  • Apple Inc. (2017) Choosing a Membership. https://developer.apple.com/support/compare-memberships/, (last visited: Mar 29, 2018)

  • Bates D, Mächler M, Bolker B, Walker S (2015) Fitting linear mixed-effects models using lme4. J Stat Softw 67(1):1–48

    Article  Google Scholar 

  • Becker R, Chernihov Y, Shavitt Y, Zilberman N (2012) An analysis of the Steam community network evolution. In: Proceedings of the 27th Convention of Electrical & Electronics Engineers in Israel (IEEEI). IEEE, pp 1–5

  • Blackburn J, Kourtellis N, Skvoretz J, Ripeanu M, Iamnitchi A (2014) Cheating in online games: a social network perspective. ACM Trans Internet Technol (TOIT) 13(3):9

    Article  Google Scholar 

  • Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet Allocation. J Mach Learn Res 3(Jan):993–1022

    MATH  Google Scholar 

  • Chambers C, Feng Wc, Sahu S, Saha D (2005) Measurement-based characterization of a collection of on-line games. In: Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement, USENIX Association, pp 1–1

  • Chen N, Lin J, Hoi SCH, Xiao X, Zhang B (2014) Ar-miner: Mining Informative reviews for developers from mobile app marketplace. In: Proceedings of the 36th International Conference on Software Engineering (ICSE). ACM, New York, pp 767–778

  • Cheung GK, Zimmermann T, Nagappan N (2014) The first hour experience: How the initial play can engage (or lose) new players. In: Proceedings of the First ACM SIGCHI Annual Symposium on Computer-human Interaction in Play (CHI PLAY). ACM, New York, pp 57–66

  • Ciurumelea A, Schaufelbühl A, Panichella S, Gall HC (2017) Analyzing reviews and code of mobile apps for better release planning. In: Proceedings of the 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, pp 91–102

  • Cobbett R (2017) From Shareware superstars to the Steam gold rush: How indie conquered the PC. http://www.pcgamer.com/from-shareware-superstars-to-the-steam-gold-rush-how-indie-conquered-the-pc/ (last visited: Mar 29 2018)

  • Coleman M, Liau TL (1975) A computer readability formula designed for machine scoring. J Appl Psychol 60(2):283

    Article  Google Scholar 

  • Di Sorbo A, Panichella S, Alexandru CV, Shimagaki J, Visaggio CA, Canfora G, Gall HC (2016) What would users change in my app? Summarizing app reviews for recommending software changes. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, pp 499–510

  • Fu B, Lin J, Li L, Faloutsos C, Hong J, Sadeh N (2013) Why people hate your app: Making sense of user feedback in a mobile app store. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp 1276–1284

  • Genc-Nayebi N, Abran A (2017) A systematic literature review: Opinion mining studies from mobile app store user reviews. J Syst Softw (JSS) 125:207–219

    Article  Google Scholar 

  • Google (2017) How to use the Play Console. https://support.google.com/googleplay/android-developer/answer/6112435?hl=en, (last visited: Mar 29, 2018)

  • Grano G, Di Sorbo A, Mercaldo F, Visaggio CA, Canfora G, Panichella S (2017) Android apps and user feedback: a dataset for software evolution and quality improvement. In: Proceedings of the 2nd ACM SIGSOFT International Workshop on App Market Analytics. ACM, pp 8–11

  • Gu X, Kim S (2015) What parts of your apps are loved by users?. In: 30Th international conference on automated software engineering (ASE). IEEE, pp 760–770

  • Hassan S, Tantithamthavorn C, Bezemer CP, Hassan AE (2018) Studying the dialogue between users and developers of free apps in the google play store. Empir Softw Eng 23(3):1275–1312

    Article  Google Scholar 

  • Hoon L, Vasa R, Schneider JG, Mouzakis K (2012) A preliminary analysis of vocabulary in mobile app user reviews. In: Proceedings of the 24th Australian Computer-Human Interaction Conference. ACM, pp 245–248

  • Iacob C, Harrison R (2013) Retrieving and analyzing mobile apps feature requests from online reviews. In: 10Th working conference on mining software repositories (MSR). IEEE, pp 41–44

  • Khalid H, Shihab E, Nagappan M, Hassan AE (2015) What do mobile app users complain about? IEEE Softw 32(3):70–77

    Article  Google Scholar 

  • Kincaid J (1975) Derivation of New Readability Formulas: (automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel Research Branch report 8–75, Chief of Naval Technical Training, Naval Air Station Memphis

  • Lewis C, Whitehead J, Wardrip-fruin N (2010) What went wrong: a taxonomy of video game bugs. In: Proceedings of the 5th International Conference on the Foundations of Digital Games (FDG). ACM, pp 108–115

  • Lin D, Bezemer CP, Hassan AE (2017a) Studying the urgent updates of popular games on the Steam platform. Empir Softw Eng 22(4):2095–2126

    Article  Google Scholar 

  • Lin D, Bezemer CP, Hassan AE (2017b) An empirical study of early access games on the Steam platform. Empirical Software Engineering 23(2):771–799

    Article  Google Scholar 

  • Long JD, Feng d, Cliff N (2003) Ordinal Analysis of Behavioral Data. Wiley, Hoboken

    Book  Google Scholar 

  • Maalej W, Nabil H (2015) Bug report, feature request, or simply praise? on automatically classifying app reviews.In: 23Rd international requirements engineering conference (RE). IEEE, pp 116–125

  • Machkovech S (2018) Nintendo: letting our fans review video games might not be a good idea. https://arstechnica.com/gaming/2018/02/nintendo-has-opened-the-doors-to-fans-game-reviews-on-nintendo-com/, (last visited: Mar 29, 2018)

  • Man Y, Gao C, Lyu MR, Jiang J (2016) Experience Report: Understanding cross-platform app issues from user reviews. In: 27Th international symposium on software reliability engineering (ISSRE). IEEE, pp 138–149

  • Martens D, Johann T (2017) On the emotion of users in app reviews. In: Proceedings of the 2nd International Workshop on Emotion Awareness in Software Engineering. IEEE Press, pp 8–14

  • Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2017) A survey of app store analysis for software engineering. IEEE Trans Softw Eng (TSE) PP(99):1–32

    Google Scholar 

  • McIlroy S, Ali N, Khalid H, Hassan AE (2016) Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews. Empir Softw Eng 21(3):1067–1106

    Article  Google Scholar 

  • McIlroy S, Shang W, Ali N, Hassan A (2017) Is it worth responding to reviews? a case study of the top free apps in the Google Play store. IEEE Softw 34 (3):64–71

    Article  Google Scholar 

  • McIntosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21 (5):2146–2189

    Article  Google Scholar 

  • Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE). ACM, New York, pp 466–476

  • Noei E, Syer MD, Zou Y, Hassan AE, Keivanloo I (2017) A study of the relation of mobile device attributes with the user-perceived quality of android apps. Empir Softw Eng 22(6):3088–3116

    Article  Google Scholar 

  • Orland K (2014) Introducing Steam Gauge: Ars Reveals Steam’s most popular games. http://arstechnica.com/gaming/2014/04/introducing-steam-gauge-ars-reveals-steams-most-popular-games/ (last visited: Mar 29 2018)

  • Pagano D, Maalej W (2013) User feedback in the appstore: An empirical study. In: 21St international requirements engineering conference (RE). IEEE, pp 125–134

  • Palomba F, Salza P, Ciurumelea A, Panichella S, Gall H, Ferrucci F, De Lucia A (2017) Recommending and localizing change requests for mobile apps based on user reviews. In: Proceedings of the 39th International Conference on Software Engineering (ICSE). IEEE Press, pp 106–117

  • Panichella S, Di Sorbo A, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can i improve my app? classifying user reviews for software maintenance and evolution. In: International conference on software maintenance and evolution (ICSME). IEEE, pp 281–290

  • Pavel Djundik MB (2016) Steam DB - Steam Database. https://steamdb.info/, (last visited: Mar 29, 2018)

  • SuperData Research (2016) Market Brief — Year in Review 2016. https://web.archive.org/web/20170702202939/https://www.superdataresearch.com/market-data/market-brief-year-in-review/, (last visited: Mar 29, 2018)

  • Romano J, Kromrey JD, Coraggio J, Skowronek J, Devine L (2006) Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’s d indices the most appropriate choices. In; Annual meeting of the Southern Association for Institutional Research

  • Ruiz IJM, Nagappan M, Adams B, Berger T, Dienst S, Hassan AE (2016) Examining the rating system used in mobile-app stores. IEEE Softw 33(6):86–92

    Article  Google Scholar 

  • Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng (TSE) 25(4):557–572

    Article  Google Scholar 

  • Seaman CB, Shull F, Regardie M, Elbert D, Feldmann RL, Guo Y, Godfrey S (2008) Defect categorization: making use of a decade of widely varying historical data. In: Proceedings of the 2nd International Symposium on Empirical Software Engineering and Measurement (ESEM). ACM, pp 149–157

  • Sergey G (2016) SteamSpy - All the data and stats about Steam games. http://steamspy.com/, (last visited: Mar 29, 2018)

  • Sifa R, Drachen A, Bauckhage C (2015) Large-scale cross-game player behavior analysis on steam. In: Artificial Intelligence and Interactive Digital Entertainment International Conference. AAAI Press, pp 198–204

  • Stern C (2012) what makes a game indie: a universal definition. http://sinisterdesign.net/what-makes-a-game-indie-a-universal-definitio n/, (last visited: Mar 29, 2018)

  • Takahashi D (2016) PwC: Game Industry to grow nearly 5% annually through 2020. https://venturebeat.com/2016/06/08/the-u-s-and-global-game-industries-will-grow-a-healthy-amount-by-2020-pwc-forecasts/ (last visited: Mar 29 2018)

  • Valve (2016a) Steam Community. http://steamcommunity.com/, (last visited: Mar 29, 2018)

  • Valve (2016b) Steam Store. http://store.steampowered.com/, (last visited: Mar 29, 2018)

  • Vasa R, Hoon L, Mouzakis K, Noguchi A (2012) A preliminary analysis of mobile app user reviews. In: Proceedings of the 24th Australian Computer-Human Interaction Conference. ACM, pp 241–244

  • Villarroel L, Bavota G, Russo B, Oliveto R, Di Penta M (2016) Release planning of mobile apps based on user reviews. In: Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM, pp 14–24

  • Washburn Jr, M, Sathiyanarayanan P, Nagappan M, Zimmermann T, Bird C (2016) What went right and what went wrong: An analysis of 155 postmortems from game development. In: Proceedings of the 38th International Conference on Software Engineering (ICSE). IEEE/ACM, pp 280–289

  • Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bullet 1 (6):80–83

    Article  Google Scholar 

Download references

Acknowledgements

We are grateful to Sergey Galyonkin, the owner of Steam Spy, who generously gave us access to all the historical data of Steam collected by Steam Spy for this research.

Author information

Authors and Affiliations

  1. Software Analysis and Intelligence Lab (SAIL), Queen’s University, Kingston, ON, Canada

    Dayi Lin, Cor-Paul Bezemer & Ahmed E. Hassan

  2. Department of Electrical and Computer Engineering, Queen’s University, Kingston, ON, Canada

    Ying Zou

Authors
  1. Dayi Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cor-Paul Bezemer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ying Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ahmed E. Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dayi Lin.

Additional information

Communicated by: Massimiliano Di Penta

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, D., Bezemer, CP., Zou, Y. et al. An empirical study of game reviews on the Steam platform. Empir Software Eng 24, 170–207 (2019). https://doi.org/10.1007/s10664-018-9627-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-018-9627-4

Keywords

Search

Navigation