Wenyu Wang
Wei Yang
Tao Xie
ABSTRACT
User Interface (UI) testing is a popular approach to ensure the quality of mobile apps. Numerous test generation tools have been developed to support UI testing on mobile apps, especially for Android apps. Previous work evaluates and compares different test generation tools using only relatively simple open-source apps, while real-world industrial apps tend to have more complex functionalities and implementations. There is no direct comparison among test generation tools with regard to effectiveness and ease-of-use on these industrial apps. To address such limitation, we study existing state-of-the-art or state-of-the-practice test generation tools on 68 widely-used industrial apps. We directly compare the tools with regard to code coverage and fault-detection ability. According to our results, Monkey, a state-of-the-practice tool from Google, achieves the highest method coverage on 22 of 41 apps whose method coverage data can be obtained. Of all 68 apps under study, Monkey also achieves the highest activity coverage on 35 apps, while Stoat, a state-of-the-art tool, is able to trigger the highest number of unique crashes on 23 apps. By analyzing the experimental results, we provide suggestions for combining different test generation tools to achieve better performance. We also report our experience in applying these tools to industrial apps under study. Our study results give insights on how Android UI test generation tools could be improved to better handle complex industrial apps.
- Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M. Memon. 2012. Using GUI Ripping for Automated Testing of Android Applications. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering (ASE 2012). ACM, New York, NY, USA, 258–261. Google Scholar
Digital Library
- Saswat Anand. 2016. ELLA: A Tool for Binary Instrumentation of Android Apps. https://github.com/saswatanand/ellaGoogle Scholar
- Saswat Anand, Mayur Naik, Mary Jean Harrold, and Hongseok Yang. 2012. Automated Concolic Testing of Smartphone Apps. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (FSE ’12). ACM, New York, NY, USA, Article 59, 11 pages. Google ScholarDigital Library
- Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. 2003. An Introduction to MCMC for Machine Learning. Machine Learning 50, 1 (01 Jan 2003), 5–43.Google Scholar
- Dana Angluin. 1987. Learning regular sets from queries and counterexamples. Information and Computation 75, 2 (1987), 87 – 106. 0890-5401(87)90052-6 Google ScholarDigital Library
- Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and Depth-first Exploration for Systematic Testing of Android Apps. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications (OOPSLA ’13). ACM, New York, NY, USA, 641–660. Google ScholarDigital Library
- Wontae Choi, George Necula, and Koushik Sen. 2013. Guided GUI Testing of Android Apps with Minimal Restart and Approximate Learning. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications (OOPSLA ’13). ACM, New York, NY, USA, 623–640. Google ScholarDigital Library
- Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet?. In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) (ASE ’15). IEEE Computer Society, Washington, DC, USA, 429–440.Google ScholarDigital Library
- K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. Trans. Evol. Comp 6, 2 (April 2002), 182–197. Google ScholarDigital Library
- Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’05). ACM, New York, NY, USA, 213–223. Google ScholarDigital Library
- Google. 2017. Android Dalvik Executable format. https://source.android.com/ devices/tech/dalvik/dex-formatGoogle Scholar
- Google. 2017. ART and Dalvik. https://source.android.com/devices/tech/dalvik/Google Scholar
- Google. 2018. Android 64K Method limit. https://developer.android.com/studio/ build/multidexGoogle Scholar
- Google. 2018. Android App Components. https://developer.android.com/guide/ components/fundamentals#ComponentsGoogle Scholar
- Google. 2018. Android Apps on Play Store. https://play.google.com/store/appsGoogle Scholar
- Google. 2018. Android Debug Bridge (adb). https://developer.android.com/ studio/command-line/adbGoogle Scholar
- Google. 2018. Android Monkey. https://developer.android.com/studio/test/ monkeyGoogle Scholar
- Google. 2018. Android Platform Architecture. https://developer.android.com/ guide/platform/Google Scholar
- Google. 2018. Logcat command-line tool. https://developer.android.com/studio/ command-line/logcatGoogle Scholar
- Xiaocong He. 2018. Python wrapper of Android uiautomator test tool. https: //github.com/xiaocong/uiautomatorGoogle Scholar
- K. Inkumsah and T. Xie. 2008. Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution. In 2008 23rd IEEE/ACM International Conference on Automated Software Engineering. 297– 306. Google ScholarDigital Library
- Business Insider. 2018. WeChat has hit 1 billion monthly active users. http://www. businessinsider.com/wechat-has-hit-1-billion-monthly-active-users-2018-3Google Scholar
- Wing Lam, Zhengkai Wu, Dengfeng Li, Wenyu Wang, Haibing Zheng, Hui Luo, Peng Yan, Yuetang Deng, and Tao Xie. 2017. Record and Replay for Android: Are We There Yet in Industrial Cases?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE ’17). ACM, New York, NY, USA, 854–859. Google ScholarDigital Library
- Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. DroidBot: A Lightweight UI-guided Test Input Generator for Android. In Proceedings of the 39th International Conference on Software Engineering Companion (ICSE-C ’17). IEEE Press, Piscataway, NJ, USA, 23–26. Google ScholarDigital Library
- Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2018. DroidBot: A lightweight test input generator for Android. https://github.com/honeynet/ droidbotGoogle Scholar
- Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An Input Generation System for Android Apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE ’13). ACM, New York, NY, USA, 224–234. Google ScholarDigital Library
- Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective Automated Testing for Android Applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA ’16). ACM, New York, NY, USA, 94–105. Google ScholarDigital Library
- Ke Mao, Mark Harman, and Yue Jia. 2017. Crowd Intelligence Enhances Automated Mobile Testing. In Proceedings of the 32Nd IEEE/ACM International Conference on Automated Software Engineering (ASE ’17). IEEE Press, Piscataway, NJ, USA, 16–26. http://dl.acm.org/citation.cfm?id=3155562.3155569 Google ScholarDigital Library
- Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, Stochastic Model-based GUI Testing of Android Apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE ’17). ACM, New York, NY, USA, 245–256. Google ScholarDigital Library
- Raja Vallée-Rai, Phong Co, Etienne Gagnon, Laurie Hendren, Patrick Lam, and Vijay Sundaresan. 1999. Soot - a Java Bytecode Optimization Framework. (1999), 13–. http://dl.acm.org/citation.cfm?id=781995.782008Google Scholar
- Wei Yang, Mukul R. Prasad, and Tao Xie. 2013. A Grey-box Approach for Automated GUI-model Generation of Mobile Applications. In Proceedings of the 16th International Conference on Fundamental Approaches to Software Engineering (FASE ’13). Springer-Verlag, Berlin, Heidelberg, 250–265. 978-3-642-37057-1_19 Google ScholarDigital Library
- Xia Zeng, Dengfeng Li, Wujie Zheng, Fan Xia, Yuetang Deng, Wing Lam, Wei Yang, and Tao Xie. 2016. Automated Test Input Generation for Android: Are We Really There Yet in an Industrial Case?. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE ’16). ACM, New York, NY, USA, 987–992. Google ScholarDigital Library
Index Terms
- An empirical study of Android test generation tools in industrial cases
Recommendations
A framework for testing Android apps by reusing test cases
MOBILESoft '19: Proceedings of the 6th International Conference on Mobile Software Engineering and SystemsAndroid apps are generally developed by an individual developer or a small team of developers, and the developers may not have experience of testing Android apps or they may not have experience of testing any software systems. Furthermore, even an ...
DroidMate-2: a platform for Android test generation
ASE '18: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software EngineeringAndroid applications (apps) represent an ever increasing portion of the software market. Automated test input generators are the state of the art for testing and security analysis. We introduce DroidMate-2 (DM-2), a platform to easily assist both ...
Seven reasons why: an in-depth study of the limitations of random test input generation for Android
ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software EngineeringExperience paper: Testing of mobile apps is time-consuming and requires a great deal of manual effort. For this reason, industry and academic researchers have proposed a number of test input generation techniques for automating app testing. Although ...
Comments