Liu Wang
Tao Zhang
Shangguang Wang
Xuanzhe Liu
ABSTRACT
The app markets enable users to submit feedback for downloaded apps in the form of star ratings and text reviews, which are meant to be helpful and trustworthy for decision making to both developers and other users. App markets have released strict guidelines/policies for user review submissions. However, there has been growing evidence showing the untrustworthy and poor-quality of app reviews, making the app store review environment a shambles. Therefore, review removal is a common practice, and market maintainers have to remove undesired reviews from the market periodically in a reactive manner. Although some reports and news outlets have mentioned removed reviews, our research community still lacks the comprehensive understanding of the landscape of this kind of reviews. To fill the void, in this paper, we present a large-scale and longitudinal study of removed reviews in iOS App Store. We first collaborate with our industry partner to collect over 30 million removed reviews for 33,665 popular apps over the course of a full year in 2020. This comprehensive dataset enables us to characterize the overall landscape of removed reviews. We next investigate the practical reasons leading to the removal of policy-violating reviews, and summarize several interesting reasons, including fake reviews, offensive reviews, etc. More importantly, most of these mis-behaviors can be reflected on reviews’ basic information including the posters, narrative content, and posting time. It motivates us to design an automated approach to flag the policy-violation reviews, and our experiment result on the labelled benchmark can achieve a good performance (F1=97%). We further make an attempt to apply our approach to the large-scale industry setting, and the result suggests the promising industry usage scenario of our approach. Our approach can act as a gatekeeper to pinpoint policy-violation reviews beforehand, which will be quite effective in improving the maintenance process of app reviews in the industrial setting.
- 2016. How to edit and delete App Store reviews on iPhone and iPad. https://theappfactor.com/how-to-edit-and-delete-app-store-reviews-on-iphone-and-ipad/
Google Scholar
- 2017. How to Work With App Store Reviews. https://splitmetrics.com/blog/app-store-reviews/
Google Scholar
- 2019. Fake App Reviews Cause App Store Concerns. https://blog.gummicube.com/2019/07/fake-app-reviews-cause-app-store-concerns
Google Scholar
- 2020. It’s 2020 and the Google Play Store still has a major fake review problem. https://www.androidauthority.com/play-store-fake-review-problem-1082191/
Google Scholar
- 2021. How to remove spam and fake iOS and Android reviews. https://appfollow.io/blog/how-to-remove-fake-and-inappropriate-app-reviews-in-the-app-store-and-google-play.
Google Scholar
- 2021. Is Apple App Store Is Ignoring Purchased Fake Reviews? How Dating Apps are Abusing This. https://goodmenproject.com/technology/is-apple-app-store-is-ignoring-purchased-fake-reviews-how-dating-apps-are-abusing-this/.
Google Scholar
- 2021. Levenshtein distance. https://en.wikipedia.org/wiki/Levenshtein_distance
Google Scholar
- 2021. scikit-learn. https://scikit-learn.org/stable/
Google Scholar
- Naila Aslam, Waheed Yousuf Ramay, Kewen Xia, and Nadeem Sarwar. 2020. Convolutional neural network based classification of app reviews. IEEE Access, 8 (2020), 185619–185628.
Google ScholarCross Ref
- Laura V Galvis Carreno and Kristina Winbladh. 2013. Analysis of user comments: an approach for software requirements evolution. In 2013 35th international conference on software engineering (ICSE). 582–591.
Google ScholarCross Ref
- Ning Chen, Jialiu Lin, Steven CH Hoi, Xiaokui Xiao, and Boshen Zhang. 2014. AR-miner: mining informative reviews for developers from mobile app marketplace. In Proceedings of the 36th international conference on software engineering. 767–778.
Google ScholarDigital Library
- Michael Crawford, Taghi M Khoshgoftaar, Joseph D Prusa, Aaron N Richter, and Hamzah Al Najada. 2015. Survey of review spam detection using machine learning techniques. Journal of Big Data, 2, 1 (2015), 1–24.
Google ScholarCross Ref
- Xian Fan, Xiaoge Li, Feihong Du, Xin Li, and Mian Wei. 2016. Apply word vectors for sentiment analysis of APP reviews. In 2016 3rd International Conference on Systems and Informatics (ICSAI). 1062–1066.
Google ScholarCross Ref
- Cuiyun Gao, Jichuan Zeng, David Lo, Chin-Yew Lin, Michael R Lyu, and Irwin King. 2018. Infar: Insight extraction from app reviews. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 904–907.
Google ScholarDigital Library
- Emitza Guzman and Walid Maalej. 2014. How do users like this feature? a fine grained sentiment analysis of app reviews. In 2014 IEEE 22nd international requirements engineering conference (RE). 153–162.
Google Scholar
- Yangyu Hu, Haoyu Wang, Ren He, Li Li, Gareth Tyson, Ignacio Castro, Yao Guo, Lei Wu, and Guoai Xu. 2020. Mobile app squatting. In Proceedings of The Web Conference 2020. 1727–1738.
Google ScholarDigital Library
- Yangyu Hu, Haoyu Wang, Li Li, Yao Guo, Guoai Xu, and Ren He. 2019. Want to earn a few extra bucks? a first look at money-making apps. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). 332–343.
Google ScholarCross Ref
- Yangyu Hu, Haoyu Wang, Yajin Zhou, Yao Guo, Li Li, Bingxuan Luo, and Fangren Xu. 2019. Dating with scambots: Understanding the ecosystem of fraudulent dating applications. IEEE Transactions on Dependable and Secure Computing.
Google ScholarCross Ref
- Apple Inc.. 2021. App Store Review Guidelines. https://developer.apple.com/app-store/review/guidelines/
Google Scholar
- Apple Inc.. 2021. Apple Media Services Terms and Conditions. https://www.apple.com/legal/internet-services/itunes/
Google Scholar
- Raymond YK Lau, SY Liao, Ron Chi-Wai Kwok, Kaiquan Xu, Yunqing Xia, and Yuefeng Li. 2012. Text mining and probabilistic language modeling for online review spam detection. ACM Transactions on Management Information Systems (TMIS), 2, 4 (2012), 1–30.
Google ScholarDigital Library
- Li Li, Tegawendé F Bissyandé, and Jacques Klein. 2019. Rebooting research on detecting repackaged android apps: Literature review and benchmark. IEEE Transactions on Software Engineering.
Google ScholarCross Ref
- Fuqi Lin, Haoyu Wang, Liu Wang, and Xuanzhe Liu. 2021. A Longitudinal Study of Removed Apps in iOS App Store. In Proceedings of the Web Conference 2021. 1435–1446.
Google ScholarDigital Library
- Yuming Lin, Tao Zhu, Xiaoling Wang, Jingwei Zhang, and Aoying Zhou. 2014. Towards online review spam detection. In Proceedings of the 23rd International Conference on World Wide Web. 341–342.
Google ScholarDigital Library
- Google LLC. 2021. Policy Center - User Ratings, Reviews, and Installs. https://support.google.com/googleplay/android-developer/answer/9898684
Google Scholar
- Walid Maalej, Zijad Kurtanović, Hadeer Nabil, and Christoph Stanik. 2016. On the automatic classification of app reviews. Requirements Engineering, 21, 3 (2016), 311–331.
Google ScholarDigital Library
- Walid Maalej and Hadeer Nabil. 2015. Bug report, feature request, or simply praise? on automatically classifying app reviews. In 2015 IEEE 23rd international requirements engineering conference (RE). 116–125.
Google Scholar
- Yichuan Man, Cuiyun Gao, Michael R Lyu, and Jiuchun Jiang. 2016. Experience report: Understanding cross-platform app issues from user reviews. In 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE). 138–149.
Google ScholarCross Ref
- Daniel Martens and Walid Maalej. 2019. Towards understanding and detecting fake reviews in app stores. Empirical Software Engineering, 24, 6 (2019), 3316–3355.
Google ScholarCross Ref
- Dennis Pagano and Walid Maalej. 2013. User feedback in the appstore: An empirical study. In 2013 21st IEEE international requirements engineering conference (RE). 125–134.
Google Scholar
- Sebastiano Panichella, Andrea Di Sorbo, Emitza Guzman, Corrado A Visaggio, Gerardo Canfora, and Harald C Gall. 2015. How can i improve my app? classifying user reviews for software maintenance and evolution. In 2015 IEEE international conference on software maintenance and evolution (ICSME). 281–290.
Google ScholarDigital Library
- Sakshi Ranjan and Subhankar Mishra. 2020. Comparative sentiment analysis of App reviews. In 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT). 1–7.
Google ScholarCross Ref
- Abbas Razaghpanah, Rishab Nithyanand, Narseo Vallina-Rodriguez, Srikanth Sundaresan, Mark Allman, Christian Kreibich, and Phillipa Gill. 2018. Apps, trackers, privacy, and regulators: A global study of the mobile tracking ecosystem.
Google Scholar
- Haoyu Wang, Hao Li, and Yao Guo. 2019. Understanding the evolution of mobile app ecosystems: A longitudinal measurement study of google play. In The World Wide Web Conference. 1988–1999.
Google ScholarDigital Library
- Haoyu Wang, Zhe Liu, Yao Guo, Xiangqun Chen, Miao Zhang, Guoai Xu, and Jason Hong. 2017. An explorative study of the mobile app ecosystem from app developers’ perspective. In Proceedings of the 26th International Conference on World Wide Web. 163–172.
Google ScholarDigital Library
- Haoyu Wang, Junjun Si, Hao Li, and Yao Guo. 2019. RmvDroid: Towards a Reliable Android Malware Dataset with App Metadata. In Proceedings of the 16th International Conference on Mining Software Repositories. 404–408.
Google ScholarDigital Library
- Sihong Xie, Guan Wang, Shuyang Lin, and Philip S Yu. 2012. Review spam detection via time series pattern discovery. In Proceedings of the 21st International Conference on World Wide Web. 635–636.
Google ScholarDigital Library
- Monali Zende and Aruna Gupta. 2016. Ranking Fraud and Fake Reviews Detection for Mobile Apps.. International Journal of Advanced Research in Computer Science, 7, 3 (2016).
Google Scholar
- Xian Zhan, Lingling Fan, Tianming Liu, Sen Chen, Li Li, Haoyu Wang, Yifei Xu, Xiapu Luo, and Yang Liu. 2020. Automated Third-Party Library Detection for Android Applications: Are We There Yet? In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). 919–930.
Google ScholarDigital Library
- Yao Zhu, Hongzhi Liu, Yingpeng Du, and Zhonghai Wu. 2021. IFSpard: An Information Fusion-based Framework for Spam Review Detection. In Proceedings of the Web Conference 2021. 507–517.
Google ScholarDigital Library
Index Terms
- Demystifying “removed reviews” in iOS app store
Recommendations
A Longitudinal Study of Removed Apps in iOS App Store
WWW '21: Proceedings of the Web Conference 2021To improve app quality and nip the potential threats in the bud, modern app markets have released strict guidelines along with app vetting process before app publishing. However, there has been growing evidence showing the ineffectiveness of app vetting,...
A Deep Dive into the Featured iOS Apps
Internetware '23: Proceedings of the 14th Asia-Pacific Symposium on InternetwareMillions of apps in markets have made it difficult for mobile users to find fancy and high quality apps. Mobile app markets have deployed mechanisms to recommend apps to users. Apple usually features apps in the iOS App Store, and mobile users could ...
Comments