Yiheng Xiong
Zhendong Su
ABSTRACT
Android apps are ubiquitous and serve many aspects of our daily lives. Ensuring their functional correctness is crucial for their success. To date, we still lack a general and in-depth understanding of functional bugs, which hinders the development of practices and techniques to tackle functional bugs. To fill this gap, we conduct the first systematic study on 399 functional bugs from 8 popular open-source and representative Android apps to investigate the root causes, bug symptoms, test oracles, and the capabilities and limitations of existing testing techniques. This study took us substantial effort. It reveals several new interesting findings and implications which help shed light on future research on tackling functional bugs. Furthermore, findings from our study guided the design of a proof-of-concept differential testing tool, RegDroid, to automatically find functional bugs in Android apps. We applied RegDroid on 5 real-world popular apps, and successfully discovered 14 functional bugs, 10 of which were previously unknown and affected the latest released versions—all these 10 bugs have been confirmed and fixed by the app developers. Specifically, 10 out of these 14 found bugs cannot be found by existing testing techniques. We have made all the artifacts (including the dataset of 399 functional bugs and RegDroid) in our work publicly available at https://github.com/Android-Functional-bugs-study/home.
- Christoffer Quist Adamsen, Gianluca Mezzetti, and Anders Møller. 2015. Systematic execution of Android test suites in adverse conditions. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA). 83–93. https://doi.org/10.1145/2771783.2771786
Google Scholar
Digital Library
- Anders Toxboe. 2022. Design patterns. https://ui-patterns.com/patterns
Google Scholar
- Android. 2022. Input events overview. https://developer.android.com/develop/ui/views/touch-and-input/input-events
Google Scholar
- AppBrain Team. 2023. Most popular Google Play categories. https://www.appbrain.com/stats/android-market-app-categories
Google Scholar
- Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Trans. Software Eng., 41, 5 (2015), 507–525. https://doi.org/10.1109/TSE.2014.2372785
Google ScholarDigital Library
- Farnaz Behrang and Alessandro Orso. 2019. Test migration between mobile apps with similar functionality. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 54–65. https://doi.org/10.1109/ASE.2019.00016
Google ScholarDigital Library
- Tsong Y. Chen, Shing C. Cheung, and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. HKUST-CS98-01, Hong Kong University of Science and Technology.
Google Scholar
- Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet? (E). In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). 429–440. https://doi.org/10.1109/ASE.2015.89
Google ScholarDigital Library
- Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP). 268–279. https://doi.org/10.1145/351240.351266
Google ScholarDigital Library
- Compuware. 2013. Users Have Low Tolerance For Buggy Apps - Only 16% Will Try A Failing App More Than Twice. https://techcrunch.com/2013/03/12/users-have-low-tolerance-for-buggy-apps-only-16-will-try-a-failing-app-more-than-twice/
Google Scholar
- Quan Chau Dong Do, Guowei Yang, Meiru Che, Darren Hui, and Jefferson Ridgeway. 2016. Redroid: A Regression Test Selection Approach for Android Applications. In The 28th International Conference on Software Engineering and Knowledge Engineering (SEKE). 486–491. https://doi.org/10.18293/SEKE2016-223
Google ScholarCross Ref
- Dawson R. Engler, David Yu Chen, and Andy Chou. 2001. Bugs as Deviant Behavior: A General Approach to Inferring Errors in Systems Code. In Proceedings of the 18th ACM Symposium on Operating System Principles (SOSP 2001). 57–72. https://doi.org/10.1145/502034.502041
Google ScholarDigital Library
- Camilo Escobar-Velásquez, Michael Osorio-Riaño, Juan Dominguez-Osorio, Maria Arevalo, and Mario Linares-Vásquez. 2020. An Empirical Study of i18n Collateral Changes and Bugs in GUIs of Android apps. In IEEE International Conference on Software Maintenance and Evolution (ICSME). 581–592. https://doi.org/10.1109/ICSME46990.2020.00061
Google ScholarCross Ref
- Lingling Fan, Ting Su, Sen Chen, Guozhu Meng, Yang Liu, Lihua Xu, Geguang Pu, and Zhendong Su. 2018. Large-scale analysis of framework-specific exceptions in Android apps. In Proceedings of the 40th International Conference on Software Engineering (ICSE). 408–419. https://doi.org/10.1145/3180155.3180222
Google ScholarDigital Library
- Umar Farooq, Zhijia Zhao, Manu Sridharan, and Iulian Neamtiu. 2020. Livedroid: Identifying and preserving mobile app state in volatile runtime environments. Proceedings of the ACM on Programming Languages, 4, OOPSLA (2020), 1–30. https://doi.org/10.1145/3428228
Google ScholarDigital Library
- Mattia Fazzini and Alessandro Orso. 2017. Automated cross-platform inconsistency detection for mobile apps. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 308–318. https://doi.org/10.1109/ASE.2017.8115644
Google ScholarCross Ref
- Mattia Fazzini, Martin Prammer, Marcelo d’Amorim, and Alessandro Orso. 2018. Automatically translating bug reports into test cases for mobile apps. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 141–152. https://doi.org/10.1145/3213846.3213869
Google ScholarDigital Library
- Gil Bouhnick. 2019. Visually distorted - when symmetrical UI looks all wrong. https://www.mobilespoon.net/2019/08/visually-distorted-when-ui-looks-all.html
Google Scholar
- Google. 2022. Google Issue Tracker. https://issuetracker.google.com/issues/37124582?pli=1
Google Scholar
- Google. 2022. Google Play Store. https://play.google.com/store/apps
Google Scholar
- Wunan Guo, Zhen Dong, Liwei Shen, Wei Tian, Ting Su, and Xin Peng. 2022. Detecting and fixing data loss issues in Android apps. In ISSTA ’22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 605–616. https://doi.org/10.1145/3533767.3534402
Google ScholarDigital Library
- Hacker News. 2013. Tell Facebook: There’s a severe bug when changing profile pics on the iOS app. Retrieved 2022-8 from https://news.ycombinator.com/item?id=6456285
Google Scholar
- Cuixiong Hu and Iulian Neamtiu. 2011. Automating GUI testing for Android applications. In Proceedings of the 6th International Workshop on Automation of Software Test (AST). 77–83. https://doi.org/10.1145/1982595.1982612
Google ScholarDigital Library
- Gang Hu, Linjie Zhu, and Junfeng Yang. 2018. AppFlow: using machine learning to synthesize robust, reusable UI tests. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT (FSE). 269–282. https://doi.org/10.1145/3236024.3236055
Google ScholarDigital Library
- Jiajun Hu, Lili Wei, Yepang Liu, Shing-Chi Cheung, and Huaxun Huang. 2018. A tale of two cities: how WebView induces bugs to Android applications. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE). 702–713. https://doi.org/10.1145/3238147.3238180
Google ScholarDigital Library
- Ajay Kumar Jha, Sunghee Lee, and Woo Jin Lee. 2019. Characterizing Android-specific crash bugs. In Proceedings of the 6th International Conference on Mobile Software Engineering and Systems (MOBILESoft). 111–122. https://doi.org/10.1109/MOBILESoft.2019.00024
Google ScholarCross Ref
- Bo Jiang, Yu Wu, Yongfei Zhang, Zhenyu Zhang, and W. K. Chan. 2018. ReTestDroid: Towards Safer Regression Test Selection for Android Application. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC). 235–244. https://doi.org/10.1109/COMPSAC.2018.00037
Google ScholarCross Ref
- Jack Johnson, Junayed Mahmud, Tyler Wendland, Kevin Moran, Julia Rubin, and Mattia Fazzini. 2022. An Empirical Investigation into the Reproduction of Bug Reports for Android Apps. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 321–322. https://doi.org/10.1109/SANER53432.2022.00048
Google ScholarCross Ref
- Yavuz Köroglu and Alper Sen. 2021. Functional test generation from UI test scenarios using reinforcement learning for android applications. Softw. Test. Verification Reliab., 31, 3 (2021), https://doi.org/10.1002/stvr.1752
Google ScholarCross Ref
- Edmund S. L. Lam, Peilun Zhang, and Bor-Yuh Evan Chang. 2017. ChimpCheck: property-based randomized test generation for interactive apps. In Proceedings of the 2017 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!). 58–77. https://doi.org/10.1145/3133850.3133853
Google ScholarDigital Library
- Xiao Li, Nana Chang, Yan Wang, Haohua Huang, Yu Pei, Linzhang Wang, and Xuandong Li. 2017. ATOM: Automatic maintenance of GUI test scripts for evolving mobile applications. In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). 161–171. https://doi.org/10.1109/ICST.2017.22
Google ScholarCross Ref
- Zhenmin Li, Lin Tan, Xuanhui Wang, Shan Lu, Yuanyuan Zhou, and Chengxiang Zhai. 2006. Have things changed now?: an empirical study of bug characteristics in modern open source software. In Proceedings of the 1st Workshop on Architectural and System Support for Improving Software Dependability (ASID). 25–33. https://doi.org/10.1145/1181309.1181314
Google ScholarDigital Library
- Jun-Wei Lin, Reyhaneh Jabbarvand, and Sam Malek. 2019. Test Transfer Across Mobile Apps Through Semantic Mapping. In 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 42–53. https://doi.org/10.1109/ASE.2019.00015
Google ScholarDigital Library
- Ying-Dar Lin, José F. Rojas, Edward T.-H. Chu, and Yuan-Cheng Lai. 2014. On the Accuracy, Efficiency, and Reusability of Automated Test Oracles for Android Devices. IEEE Trans. Software Eng., 40, 10 (2014), 957–970. https://doi.org/10.1109/TSE.2014.2331982
Google ScholarCross Ref
- Lint Team. 2022. Android Lint. http://tools.android.com/lint/overview
Google Scholar
- Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020. Owl Eyes: Spotting UI Display Issues via Visual Understanding. In 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). 398–409. https://doi.org/10.1145/3324884.3416547
Google ScholarDigital Library
- Localytics. 2019. 25% of Users Abandon Apps After One Use. http://info.localytics.com/blog/25-of-users-abandon-apps-after-one-use
Google Scholar
- Leonardo Mariani, Mauro Pezzè, and Daniele Zuddas. 2018. Augusto: exploiting popular functionalities for the generation of semantic GUI tests with Oracles. In Proceedings of the 40th International Conference on Software Engineering (ICSE). 280–290. https://doi.org/10.1145/3180155.3180162
Google ScholarDigital Library
- Martijn van Welie. 2008. Pattern library. http://www.welie.com/patterns/index.php
Google Scholar
- William M McKeeman. 1998. Differential testing for software. Digital Technical Journal, 10, 1 (1998), 100–107.
Google Scholar
- Motherboard. 2020. Here’s the Shadow Inc. App That Failed in Iowa Last Night. https://www.vice.com/en_us/article/y3m33x/heres-the-shadow-inc-app-that-failed-in-iowa-last-night
Google Scholar
- Liam O’Connor and Oskar Wickström. 2022. Quickstrom: property-based acceptance testing with LTL specifications. In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI). 1025–1038. https://doi.org/10.1145/3519939.3523728
Google ScholarDigital Library
- Dominik Pacholczyk. 2014. Mobile UI Design Patterns A Deeper Look At The Hottest Apps Today. www.uxpin.com.
Google Scholar
- Reuters. 2021. Japan’s COVID-19 app failed to pass on some contact warnings. https://www.reuters.com/article/us-health-coronavirus-japan-app-idUSKBN2A31BA
Google Scholar
- Vincenzo Riccio, Domenico Amalfitano, and Anna Rita Fasolino. 2018. Is this the lifecycle we really want?: an automated black-box testing approach for Android activities. In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops (ISSTA). 68–77. https://doi.org/10.1145/3236454.3236490
Google ScholarDigital Library
- Oliviero Riganelli, Simone Paolo Mottadelli, Claudio Rota, Daniela Micucci, and Leonardo Mariani. 2020. Data loss detector: automatically revealing data loss bugs in Android apps. In ISSTA ’20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 141–152. https://doi.org/10.1145/3395363.3397379
Google ScholarDigital Library
- Ariel Rosenfeld, Odaya Kardashov, and Orel Zang. 2018. Automation of android applications functional testing using machine learning activities classification. In Proceedings of the 5th International Conference on Mobile Software Engineering and Systems. 122–132. https://doi.org/10.1145/3197231.3197241
Google ScholarDigital Library
- Konstantin Rubinov and Luciano Baresi. 2018. What Are We Missing When Testing Our Android Apps? Computer, 51, 4 (2018), 60–68.
Google ScholarCross Ref
- Zhiyong Shan, Tanzirul Azim, and Iulian Neamtiu. 2016. Finding resume and restart errors in android applications. ACM SIGPLAN Notices, 51, 10 (2016), 864–880. https://doi.org/10.1145/3022671.2984011
Google ScholarDigital Library
- Aman Sharma and Rupesh Nasre. 2019. QADroid: regression event selection for Android applications. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 66–77. https://doi.org/10.1145/3293882.3330550
Google ScholarDigital Library
- Sixth Tone. 2019. E-Commerce App Loses ‘Tens of Millions’ From Coupon Glitches. https://www.sixthtone.com/news/1003483/e-commerce-app-loses-tens-of-millions-from-coupon-glitches
Google Scholar
- Donna Spencer and T Warfel. 2004. Card Sorting. Boxes and arrows, 7 (2004).
Google Scholar
- statcounter. 2022. Mobile Operating System Market Share Worldwide. https://gs.statcounter.com/os-market-share/mobile/worldwide
Google Scholar
- Ting Su, Lingling Fan, Sen Chen, Yang Liu, Lihua Xu, Geguang Pu, and Zhendong Su. 2022. Why My App Crashes? Understanding and Benchmarking Framework-Specific Exceptions of Android Apps. IEEE Trans. Software Eng., 48, 4 (2022), 1115–1137. https://doi.org/10.1109/TSE.2020.3013438
Google ScholarCross Ref
- Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, Stochastic Model-based GUI Testing of Android Apps. In The joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE). 245–256. https://doi.org/10.1145/3106237.3106298
Google ScholarDigital Library
- Ting Su, Jue Wang, and Zhendong Su. 2021. Benchmarking automated GUI testing for Android against real-world bugs. In 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 119–130. https://doi.org/10.1145/3468264.3468620
Google ScholarDigital Library
- Ting Su, Yichen Yan, Jue Wang, Jingling Sun, Yiheng Xiong, Geguang Pu, Ke Wang, and Zhendong Su. 2021. Fully automated functional fuzzing of Android apps for detecting non-crashing logic bugs. Proc. ACM Program. Lang., 5, OOPSLA (2021), 1–31. https://doi.org/10.1145/3485533
Google ScholarDigital Library
- Jingling Sun, Ting Su, Junxin Li, Zhen Dong, Geguang Pu, Tao Xie, and Zhendong Su. 2021. Understanding and finding system setting-related defects in Android apps. In ISSTA ’21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 204–215. https://doi.org/10.1145/3460319.3464806
Google ScholarDigital Library
- Jingling Sun, Ting Su, Kai Liu, Chao Peng, Zhao Zhang, Geguang Pu, Tao Xie, and Zhendong Su. 2023. Characterizing and Finding System Setting-Related Defects in Android Apps. IEEE Transactions on Software Engineering, https://doi.org/10.1109/TSE.2023.3236449
Google ScholarDigital Library
- Lin Tan, Chen Liu, Zhenmin Li, Xuanhui Wang, Yuanyuan Zhou, and Chengxiang Zhai. 2014. Bug characteristics in open source software. Empirical software engineering, 19, 6 (2014), 1665–1705. https://doi.org/10.1007/s10664-013-9258-8
Google ScholarDigital Library
- ASCII Team. 2022. ASCII Codes Table. https://ascii.cl/
Google Scholar
- Jenifer Tidwell. 2010. Designing interfaces: Patterns for effective interaction design. " O’Reilly Media, Inc.".
Google ScholarDigital Library
- uiautomator2 Team. 2021. uiautomator2. https://github.com/openatx/uiautomator2
Google Scholar
- Mario Linares Vásquez, Kevin Moran, and Denys Poshyvanyk. 2017. Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). 399–410. https://doi.org/10.1109/ICSME.2017.27
Google ScholarCross Ref
- Jue Wang, Yanyan Jiang, Ting Su, Shaohua Li, Chang Xu, Jian Lu, and Zhendong Su. 2022. Detecting non-crashing functional bugs in Android apps via deep-state differential analysis. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 434–446. https://doi.org/10.1145/3540250.3549170
Google ScholarDigital Library
- Wenyu Wang, Dengfeng Li, Wei Yang, Yurui Cao, Zhenwen Zhang, Yuetang Deng, and Tao Xie. 2018. An empirical study of Android test generation tools in industrial cases. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE). 738–748. https://doi.org/10.1145/3238147.3240465
Google ScholarDigital Library
- Lili Wei, Yepang Liu, and Shing-Chi Cheung. 2016. Taming Android fragmentation: characterizing and detecting compatibility issues for Android apps. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). 226–237. https://doi.org/10.1145/2970276.2970312
Google ScholarDigital Library
- Tyler Wendland, Jingyang Sun, Junayed Mahmud, S. M. Hasan Mansur, Steven Huang, Kevin Moran, Julia Rubin, and Mattia Fazzini. 2021. Andror2: A Dataset of Manually-Reproduced Bug Reports for Android apps. In 18th IEEE/ACM International Conference on Mining Software Repositories (MSR0). 600–604. https://doi.org/10.1109/MSR52588.2021.00082
Google ScholarCross Ref
- Razieh Nokhbeh Zaeem, Mukul R. Prasad, and Sarfraz Khurshid. 2014. Automated Generation of Oracles for Testing User-Interaction Features of Mobile Apps. In Seventh IEEE International Conference on Software Testing, Verification and Validation (ICST). 183–192. https://doi.org/10.1109/ICST.2014.31
Google ScholarDigital Library
- Yu Zhao, Tingting Yu, Ting Su, Yang Liu, Wei Zheng, Jingzhi Zhang, and William G. J. Halfond. 2019. ReCDroid: automatically reproducing Android application crashes from bug reports. In Proceedings of the 41st International Conference on Software Engineering (ICSE). 128–139. https://doi.org/10.1109/ICSE.2019.00030
Google ScholarDigital Library
Index Terms
- An Empirical Study of Functional Bugs in Android Apps
Recommendations
Fully automated functional fuzzing of Android apps for detecting non-crashing logic bugs
Android apps are GUI-based event-driven software and have become ubiquitous in recent years. Obviously, functional correctness is critical for an app’s success. However, in addition to crash bugs, non-crashing functional bugs (in short as “non-crashing ...
An empirical study of Android behavioural code smells detection
AbstractMobile applications (apps) are developed quickly and evolve continuously. Each development iteration may introduce poor design choices, and therefore produce code smells. Code smells complexify source code and may impede the evolution and ...
A tale of two cities: how WebView induces bugs to Android applications
ASE '18: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software EngineeringWebView is a widely used Android component that augments a native app with web browser capabilities. It eases the interactions between an app’s native code and web code. However, the interaction mechanism of WebView induces new types of bugs in Android ...
Comments