Skip to main content
SpringerLink
Log in

Studying the consistency of star ratings and the complaints in 1 & 2-star user reviews for top free cross-platform Android and iOS apps

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

How users rate a mobile app via star ratings and user reviews is of utmost importance for the success of an app. Recent studies and surveys show that users rely heavily on star ratings and user reviews that are provided by other users, for deciding which app to download. However, understanding star ratings and user reviews is a complicated matter, since they are influenced by many factors such as the actual quality of the app and how the user perceives such quality relative to their expectations, which are in turn influenced by their prior experiences and expectations relative to other apps on the platform (e.g., iOS versus Android). Nevertheless, star ratings and user reviews provide developers with valuable information for improving the overall impression of their app. In an effort to expand their revenue and reach more users, app developers commonly build cross-platform apps, i.e., apps that are available on multiple platforms. As star ratings and user reviews are of such importance in the mobile app industry, it is essential for developers of cross-platform apps to maintain a consistent level of star ratings and user reviews for their apps across the various platforms on which they are available. In this paper, we investigate whether cross-platform apps achieve a consistent level of star ratings and user reviews. We manually identify 19 cross-platform apps and conduct an empirical study on their star ratings and user reviews. By manually tagging 9,902 1 & 2-star reviews of the studied cross-platform apps, we discover that the distribution of the frequency of complaint types varies across platforms. Finally, we study the negative impact ratio of complaint types and find that for some apps, users have higher expectations on one platform. All our proposed techniques and our methodologies are generic and can be used for any app. Our findings show that at least 79% of the studied cross-platform apps do not have consistent star ratings, which suggests that different quality assurance efforts need to be considered by developers for the different platforms that they wish to support.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Note that we removed one review from the iOS snapshots (see Section 3.2).

References

  • Akdeniz (2014) Google play crawler JAVA API. https://github.com/Akdeniz/google-play-crawler, (last visited: May 30, 2017)

  • Ali M, Joorabchi M, Mesbah A (2017) Same app, different app stores: a comparative study. In: Proceedings of the IEEE/ACM international conference on mobile software engineering and systems (mobilesoft). IEEE Computer Society, p 12

  • Android Authority (2017) Android 7 nougat update tracker. http://www.androidauthority.com/android-7-0-update-679175/, (last visited: May 30, 2017)

  • APK4Fun (2016) Version history of the app facebook in Android. http://www.apk4fun.com/history/2430/, (last visited: May 30, 2017)

  • AppAnnie (2015) U.s version of the top 50 apps chart of the app store. https://www.appannie.com/apps/ios/top/, (last visited: May 30, 2017)

  • Apple (2008) RSS Feed provided by apple for the app facebook. https://itunes.apple.com/us/rss/customerreviews/id=284882215/page=1/json, (last visited: May 30, 2017)

  • Apple (2016) If an app you installed unexpectedly quits, stops responding, or won’t open. https://support.apple.com/en-nz/HT201398, (last visited: May 30, 2017)

  • Apple (2017) App store review guidelines. https://developer.apple.com/app-store/review/guidelines/, (last visited: May 30, 2017)

  • Bavota G, Linares-Vásquez M, Bernal-Cárdenas CE, Penta MD, Oliveto R, Poshyvanyk D (2015) The impact of API change- and fault-proneness on the user ratings of Android apps. IEEE Trans Softw Eng (TSE) 41(4):384–407

    Article  Google Scholar 

  • Benenson Z, Gassmann F, Reinfelder L (2013) Android and iOS users’ differences concerning security and privacy. In: Extended abstracts on human factors in computing systems (CHI), pp 817–822

  • Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res (JMLR) 3:993–1022

    MATH  Google Scholar 

  • Chen Y, Xu H, Zhou Y, Zhu S (2013) Is this app safe for children?: a comparison study of maturity ratings on Android and iOS applications. In: Proceedings of the 22nd international conference on world wide web (WWW). ACM, New York, pp 201–212

  • Chen N, Lin J, Hoi SCH, Xiao X, Zhang B (2014) AR-Miner: mining informative reviews for developers from mobile app marketplace. In: Proceedings of the 36th international conference on software engineering (ICSE). ACM, New York, pp 767–778

  • Dalmasso I, Datta SK, Bonnet C, Nikaein N (2013) Survey, comparison and evaluation of cross platform mobile application development tools. In: 9th international wireless communications and mobile computing conference (IWCMC), pp 323–328

  • Dann J (2012) Under the hood: Rebuilding facebook for iOS. https://www.facebook.com/notes/facebook-engineering/under-the-hood-rebuilding-facebook-for-ios/10151036091753920/, (last visited: May 30, 2017)

  • Di Sorbo A, Panichella S, Alexandru CV, Shimagaki J, Visaggio CA, Canfora G, Gall HC (2016) What would users change in my app? Summarizing app reviews for recommending software changes. In: Proceedings of the 24th ACM SIGSOFT international symposium on foundations of software engineering (FSE), ACM, pp 499–510

  • Fu B, Lin J, Li L, Faloutsos C, Hong J, Sadeh N (2013) Why people hate your app: making sense of user feedback in a mobile app store. In: Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining (KDD). ACM, New York, pp 1276–1284

  • Graphpad Software (2015) Interpreting results: skewness and kurtosis. http://www.graphpad.com/guides/prism/6/statistics/index.htm?stat_skewness_and_kurtosis.htm, (last visited: May 30, 2017)

  • Guzman E, Maalej W (2014) How do users like this feature? a fine grained sentiment analysis of app reviews. In: 22nd international requirements engineering conference (RE). IEEE, pp 153–162

  • Harman M, Jia Y, Zhang Y (2012) App store mining and analysis: MSR for app stores. In: 9th working conference on mining software repositories (MSR). IEEE, pp 108–111

  • Hartmann G, Stead G, DeGani A (2011) Cross-platform mobile development. Mobile Learning Environment, Cambridge

    Google Scholar 

  • Hassan S, Shang W, Hassan AE (2017) An empirical study of emergency updates for top Android mobile apps. Empir Softw Eng (EMSE) 22(1):505–546

    Article  Google Scholar 

  • Heitkötter H, Hanschke S, Majchrzak TA (2013) Evaluating cross-platform development approaches for mobile applications. In: Web information systems and technologies, springer, pp 120–138

    Google Scholar 

  • Herraiz I, Shihab E, Nguyen THD, Hassan AE (2011) Impact of installation counts on perceived quality: a case study on Debian. In: 18th working conference on reverse engineering (WCRE), pp 219–228

  • Hixon T (2014) What kind of person prefers an iphone? https://www.forbes.com/sites/toddhixon/2014/04/10/what-kind-of-person-prefers-an-iphone/, (last visited: May 30, 2017)

  • Howison J, Crowston K (2004) The perils and pitfalls of mining SourceForge. In: International workshop on mining software repositories (MSR), pp 7–11

  • Hu H, Bezemer CP, Hassan AE (2017) Supplementary material: Studying the consistency of star ratings and 1 & 2-star user reviews for top free cross-platform Android and iOS apps. http://sailhome.cs.queensu.ca/replication/cross-platform-mobile-apps/, (last visited: May 30, 2017)

  • Joorabchi M, Mesbah A, Kruchten P (2013) Real challenges in mobile app development. In: International symposium on empirical software engineering and measurement (ESEM). IEEE/ACM, pp 15–24

  • Joorabchi ME, Ali M, Mesbah A (2015) Detecting inconsistencies in multi-platform mobile apps. In: 26th international symposium on software reliability engineering (ISSRE). IEEE, pp 450–460

  • Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining GitHub. In: Proceedings of the 11th working conference on mining software repositories, MSR 2014, pp 92–101

  • Khalid H, Nagappan M, Shihab E, Hassan AE (2014) Prioritizing the devices to test your app on: A case study of Android game apps. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering (FSE). ACM, pp 610–620

  • Khalid H, Shihab E, Nagappan M, Hassan AE (2015) What do mobile app users complain about?. IEEE Softw 32(3):70–77

    Article  Google Scholar 

  • Kim HW, Lee HL, Son JE (2011) An exploratory study on the determinants of smartphone app purchase. In: The 11th international DSI and the 16th APDSI joint meeting, Taipei, Taiwan

  • Laurence G, Janessa R (2015) Gartner says smartphone sales surpassed one billion units in 2014. http://www.gartner.com/newsroom/id/2996817, (last visited: May 30, 2017)

  • Lim SL, Bentley PJ, Kanakam N, Ishikawa F, Honiden S (2015) Investigating country differences in mobile app user behavior and challenges for software engineering. IEEE Trans Softw Eng (TSE) 41(1):40–64

    Article  Google Scholar 

  • Linares-Vásquez M, Bavota G, Bernal-Cárdenas C, Di Penta M, Oliveto R, Poshyvanyk D (2013) API Change and fault proneness: a threat to the success of Android apps. In: Proceedings of the 9th joint meeting on foundations of software engineering (ESEC/FSE). ACM, pp 477–487

  • Long JD, Feng D, Cliff N (2003) Ordinal analysis of behavioral data. Wiley, New York

    Book  Google Scholar 

  • Martin W, Harman M, Jia Y, Sarro F, Zhang Y (2015) The app sampling problem for app store mining. In: Proceedings of the 12th working conference on mining software repositories, MSR ’15. IEEE Press, Piscataway, pp 123–133

  • Martin W, Sarro F, Harman M (2016) Causal impact analysis for app releases in google play. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2016. ACM, New York, pp 435–446. https://doi.org/10.1145/2950290.2950320

  • Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2017) A survey of app store analysis for software engineering. IEEE Trans Softw Eng (TSE) PP(99):1–32

    Google Scholar 

  • McIlroy S, Ali N, Khalid H, E Hassan A (2015) Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews. Empir Softw Eng (EMSE) 21(3):1067–1106

    Article  Google Scholar 

  • McIlroy S, Ali N, Khalid H, E Hassan A (2016) Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews. Empir Softw Eng (EMSE) 21(3):1067–1106

    Article  Google Scholar 

  • Mercado IT, Munaiah N, Meneely A (2016) The impact of cross-platform development approaches for mobile applications from the user’s perspective. In: Proceedings of the international workshop on app market analytics (WAMA). ACM, pp 43–49

  • Morani L (2015) There are now more than 24,000 different Android devices. http://qz.com/472767/there-are-now-more-than-24000-different-Android-devices/, (last visited: May 30, 2017)

  • Mudambi SM, Schuff D (2010) What makes a helpful online review? a study of customer reviews on amazon.com. MIS Q 34(1):185–200

    Article  Google Scholar 

  • Nayebi M, Adams B, Ruhe G (2016) Release practices for mobile apps – what do users and developers think?. In: 23Nd international conference on software analysis, evolution and reengineering (SANER), vol 1. IEEE, pp 552–562

  • Noei E, Syer MD, Zou Y, Hassan AE, Keivanloo I (2017) A study of the relation of mobile device attributes with the user-perceived quality of Android apps. Empir Softw Eng (EMSE) 22(6):3088–3116

    Article  Google Scholar 

  • Pagano D, Maalej W (2013) User feedback in the appstore: an empirical study. In: 21st international requirements engineering conference (RE). IEEE, pp 125–134

  • Palmieri M, Singh I, Cicchetti A (2012) Comparison of cross-platform mobile development tools. In: 16th international conference on intelligence in next generation networks (ICIN). IEEE, pp 179–186

  • Palomba F, Linares-Vásquez M, Bavota G, Oliveto R, Penta MD, Poshyvanyk D, Lucia AD (2015) User reviews matter! tracking crowdsourced reviews to support evolution of successful apps. In: 2015 IEEE international conference on software maintenance and evolution (ICSME), pp 291–300

  • Palomba F, Salza P, Ciurumelea A, Panichella S, Gall H, Ferrucci F, De Lucia A (2017) Recommending and localizing change requests for mobile apps based on user reviews. In: Proceedings of the 39th international conference on software engineering (ICSE). IEEE Press, Piscataway, pp 106–117

  • Panichella S, Sorbo AD, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can I improve my app? Classifying user reviews for software maintenance and evolution. In: International conference on software maintenance and evolution (ICSME). IEEE, pp 281–290

  • Patterson B (2016) Blake’s iOS device specifications grid. http://blakespot.com/ios_device_specifications_grid.html, (last visited: May 30, 2017)

  • Pettey C, Rob van der M (2012) Gartner says free apps will account for nearly 90 percent of total mobile app store downloads in 2012. http://www.gartner.com/newsroom/id/2153215, (last visited: May 30, 2017)

  • Porter MF (1997) An algorithm for suffix stripping. In: Readings in information retrieval. Morgan Kaufmann Publishers Inc., San Francisco, pp 313–316

  • Poschenrieder M (2015) 77% Will not download a retail app rated lower than 3 stars. https://blog.testmunk.com/77-will-not-download-a-retail-app-rated-lower-than-3-stars/, (last visited: May 30, 2017)

  • Ramon L, Ryan R, Kathy N (2015a) Number of apps available in leading app stores as of july 2015. http://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/, (last visited: May 30, 2017)

  • Ramon L, Ryan R, Kathy N (2015b) Smartphone OS market share, 2015 q2. http://www.idc.com/prodserv/smartphone-os-market-share.jsp, (last visited: May 30, 2017)

  • Raphael J (2016) Android upgrade report card: grading the manufacturers on marshmallow. http://www.computerworld.com/article/3052937/android/android-upgrade-report-card-marshmallow.html, (last visited: May 30, 2017)

  • Romano J, Kromrey JD, Coraggio J, Skowronek J, Devine L (2006) Exploring methods for evaluating group differences on the NSSE and other surveys: are the t-test and Cohen’s d indices the most appropriate choices. In: Annual meeting of the southern association for institutional research

  • Schick S (2014) Report: iOS app users are often richer than Android users. http://www.fiercewireless.com/developer/report-ios-app-users-are-often-richer-than-android-users, (last visited: May 30, 2017)

  • Shaw H, Ellis DA, Kendrick LR, Ziegler F, Wiseman R (2016) Predicting smartphone operating system from personality and individual differences. Cyberpsychol Behav Soc Netw 19(12):727–732

    Article  Google Scholar 

  • Syer MD, Nagappan M, Adams B, Hassan AE (2015) Studying the relationship between source code quality and mobile platform dependence. Software Quality Journal (SQJ) 23(3):485–508

    Article  Google Scholar 

  • Tian Y, Nagappan M, Lo D, Hassan AE (2015) What are the characteristics of high-rated apps? A case study on free Android applications. In: IEEE international conference on software maintenance and evolution (ICSME), pp 301–310

  • Villarroel L, Bavota G, Russo B, Oliveto R, Di Penta M (2016) Release planning of mobile apps based on user reviews. In: Proceedings of the 38th international conference on software engineering (ICSE). ACM, New York, pp 14–24

  • Yichuan M, Cuiyun G, Michael RL, Jiuchun J (2016) Experience report: understanding cross-platform app issues from user reviews. In: 27th international symposium on software reliability engineering (ISSRE). IEEE

  • Zaki MJ (2000) Scalable algorithms for association mining. IEEE Trans Knowl Data Eng (TKDE) 12(3):372–390

    Article  Google Scholar 

  • Zhao WX, Jiang J, Weng J, He J, Lim EP, Yan H, Li X (2011) Comparing twitter and traditional media using topic models. Springer, Berlin, pp 338–349

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Queen’s University, Kingston, ON, Canada

    Hanyang Hu, Cor-Paul Bezemer & Ahmed E. Hassan

Authors
  1. Hanyang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cor-Paul Bezemer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed E. Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cor-Paul Bezemer.

Additional information

Communicated by: Andrian Marcus

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, H., Bezemer, CP. & Hassan, A.E. Studying the consistency of star ratings and the complaints in 1 & 2-star user reviews for top free cross-platform Android and iOS apps. Empir Software Eng 23, 3442–3475 (2018). https://doi.org/10.1007/s10664-018-9604-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-018-9604-y

Keywords

Search

Navigation