skip to main content
10.1145/3106237.3106298acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Guided, stochastic model-based GUI testing of Android apps

Published: 21 August 2017 Publication History

Abstract

Mobile apps are ubiquitous, operate in complex environments and are developed under the time-to-market pressure. Ensuring their correctness and reliability thus becomes an important challenge. This paper introduces Stoat, a novel guided approach to perform stochastic model-based testing on Android apps. Stoat operates in two phases: (1) Given an app as input, it uses dynamic analysis enhanced by a weighted UI exploration strategy and static analysis to reverse engineer a stochastic model of the app's GUI interactions; and (2) it adapts Gibbs sampling to iteratively mutate/refine the stochastic model and guides test generation from the mutated models toward achieving high code and model coverage and exhibiting diverse sequences. During testing, system-level events are randomly injected to further enhance the testing effectiveness.
Stoat was evaluated on 93 open-source apps. The results show (1) the models produced by Stoat cover 17~31% more code than those by existing modeling tools; (2) Stoat detects 3X more unique crashes than two state-of-the-art testing tools, Monkey and Sapienz. Furthermore, Stoat tested 1661 most popular Google Play apps, and detected 2110 previously unknown and unique crashes. So far, 43 developers have responded that they are investigating our reports. 20 of reported crashes have been confirmed, and 8 already fixed.

References

[1]
Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M. Memon. 2012. Using GUI ripping for automated testing of Android applications. In IEEE/ACM International Conference on Automated Software Engineering, ASE’12, Essen, Germany, September 3-7, 2012. 258–261.
[2]
Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Bryan Dzung Ta, and Atif M. Memon. 2015. MobiGUITAR: Automated Model-Based Testing of Mobile Apps. IEEE Software 32, 5 (2015), 53–59.
[3]
Saswat Anand. 2017. ELLA. (2017). Retrieved 2017-2-18 from https://github.com/ saswatanand/ella
[4]
Saswat Anand, Mayur Naik, Mary Jean Harrold, and Hongseok Yang. 2012. Automated concolic testing of smartphone apps. In 20th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-20), SIGSOFT/FSE’12, Cary, NC, USA - November 11 - 16, 2012. 59.
[5]
Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. 2003. An Introduction to MCMC for Machine Learning. Machine Learning 50, 1 (2003), 5–43.
[6]
Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of Android apps. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2013, part of SPLASH 2013, Indianapolis, IN, USA, October 26-31, 2013. 641–660.
[7]
Young Min Baek and Doo-Hwan Bae. 2016. Automated model-based Android GUI testing using multi-level GUI comparison criteria. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, Singapore, September 3-7, 2016. 238–249.
[8]
Alexandre Bartel, Jacques Klein, Martin Monperrus, and Yves Le Traon. 2012. Dexpler: Converting Android Dalvik Bytecode to Jimple for Static Analysis with Soot. In ACM Sigplan International Workshop on the State Of The Art in Java Program Analysis.
[9]
Pavol Bielik, Veselin Raychev, and Martin T. Vechev. 2015. Scalable race detection for Android applications. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SPLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015. 332– 348.
[10]
Penelope A. Brooks and Atif M. Memon. 2007. Automated GUI testing guided by usage profiles. In 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007), November 5-9, 2007, Atlanta, Georgia, USA. 333–342.
[11]
Kai Chen, Peng Liu, and Yingjun Zhang. 2014. Achieving Accuracy and Scalability Simultaneously in Detecting Application Clones on Android Markets. In 36th International Conference on Software Engineering, ICSE. 175–186.
[12]
Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. 2016. Coverage-Directed Differential Testing of JVM Implementations. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation.
[13]
Yuting Chen and Zhendong Su. 2015. Guided differential testing of certificate validation in SSL/TLS implementations. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015. 793–804.
[14]
Siddhartha Chib and Edward Greenberg. 1995. Understanding the Metropolis-Hastings Algorithm. (1995).
[15]
Wontae Choi, George C. Necula, and Koushik Sen. 2013. Guided GUI testing of Android apps with minimal restart and approximate learning. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2013, part of SPLASH 2013, Indianapolis, IN, USA, October 26-31, 2013. 623–640.
[16]
Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet? (E). In 30th IEEE/ACM International Conference on Automated Software Engineering, ASE 2015, Lincoln, NE, USA, November 9-13, 2015. 429–440.
[17]
S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott, G. C. Patton, and B. M. Horowitz. 1999. Model-based Testing in Practice. In Proceedings of the 21st International Conference on Software Engineering (ICSE ’99). ACM, New York, NY, USA, 285–294.
[18]
Guilherme de Cleva Farto and Andre Takeshi Endo. 2015. Evaluating the modelbased testing approach in the context of mobile applications. Electronic notes in Theoretical computer science 314 (2015), 3–21.
[19]
Android GUITAR Developers. 2017. Android GUITAR. (2017). Retrieved 2017-2- 18 from http://sourceforge.net/apps/mediawiki/guitar/index.php?title=Android_ GUITAR
[20]
Bites Developers. 2017. Bites. (2017). Retrieved 2017-2-18 from https://code. google.com/archive/p/bites-android/
[21]
Mileage Developers. 2017. Mileage. (2017). Retrieved 2017-2-18 from https: //github.com/evancharlton/android-mileage
[22]
Soot Developers. 2017. Soot. (2017). Retrieved 2017-2-18 from https://github. com/Sable/soot
[23]
Arilo C Dias Neto, Rajesh Subramanyan, Marlon Vieira, and Guilherme H Travassos. 2007. A survey on model-based testing approaches: a systematic review. In Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007. ACM, 31–36.
[24]
Ting Su et al. 2017. Stoat. (2017). Retrieved 2017-2-18 from https://tingsu.github. io/files/stoat.html
[25]
W.R. Gilks, S. Richardson, and D. Spiegelhalter. 1995. Markov Chain Monte Carlo in Practice. Taylor & Francis. http://books.google.com/books?id=TRXrMWY_i2IC
[26]
Google. 2017. Android Logcat. (2017). Retrieved 2017-2-18 from https://developer. android.com/studio/command-line/logcat.html
[27]
Google. 2017. Android UI Automator. (2017). Retrieved 2017-2-18 from http: //developer.android.com/tools/help/uiautomator/index.html
[28]
Google. 2017. Monkey. (2017). Retrieved 2017-2-18 from http://developer.android. com/tools/help/monkey.html
[29]
AppBrain Group. 2017. AppBrain. (2017). Retrieved 2017-2-18 from http://www. appbrain.com/stats/
[30]
F-droid Group. 2017. F-Droid. (2017). Retrieved 2017-2-18 from https://f-droid. org/
[31]
Vignir Gudmundsson, Mikael Lindvall, Luca Aceto, Johann Bergthorsson, and Dharmalingam Ganesan. 2016. Model-based Testing of Mobile Systems - An Empirical Study on QuizUp Android App. In Proceedings First Workshop on Pre- and Post-Deployment Verification Techniques, PrePost@IFM 2016, Reykjavík, Iceland, 4th June 2016.
[32]
[33]
Shuai Hao, Bin Liu, Suman Nath, William G.J. Halfond, and Ramesh Govindan. 2014. PUMA: Programmable UI-automation for Large-scale Dynamic Analysis of Mobile Apps. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys ’14). ACM, New York, NY, USA, 204–217.
[34]
Xiaocong He. 2017. Python wrapper of Android UIAutomator test tool. (2017). Retrieved 2017-2-18 from https://github.com/xiaocong/uiautomator
[35]
Robert M. Hierons and Mercedes G. Merayo. 2009. Mutation testing from probabilistic and stochastic finite state machines. Journal of Systems and Software 82, 11 (2009), 1804–1818.
[36]
Gang Hu, Xinhao Yuan, Yang Tang, and Junfeng Yang. 2014. Efficiently, effectively detecting mobile app bugs with AppDoctor. In Ninth Eurosys Conference 2014, EuroSys 2014, Amsterdam, The Netherlands, April 13-16, 2014. 18:1–18:15.
[37]
Casper Svenning Jensen, Mukul R. Prasad, and Anders Møller. 2013. Automated testing with targeted event sequence generation. In International Symposium on Software Testing and Analysis, ISSTA ’13, Lugano, Switzerland, July 15-20, 2013. 67–77.
[38]
Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding deep compiler bugs via guided stochastic program mutation. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015. 386–399.
[39]
Peng Liu, Xiangyu Zhang, Marco Pistoia, Yunhui Zheng, Manoel Marques, and Lingfei Zeng. 2017. Automatic Text Input Generation for Mobile Testing. In Proceedings of the 39th International Conference on Software Engineering (ICSE ’17). IEEE Press, Piscataway, NJ, USA, 643–653.
[40]
Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: an input generation system for Android apps. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE’13, Saint Petersburg, Russian Federation, August 18-26, 2013. 224–234.
[41]
Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. EvoDroid: segmented evolutionary testing of Android apps. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014. 599–609. ESEC/FSE’17, September 4–8, 2017, Paderborn, Germany T. Su, G. Meng, Y. Chen, K. Wu, W. Yang, Y. Yao, G. Pu, Y. Liu, and Z. Su
[42]
Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbrücken, Germany, July 18-20, 2016. 94–105.
[43]
Atif M. Memon, Ishan Banerjee, and Adithya Nagarajan. 2003. GUI Ripping: Reverse Engineering of Graphical User Interfaces for Testing. In 10th Working Conference on Reverse Engineering, WCRE 2003, Victoria, Canada, November 13-16, 2003. 260–269.
[44]
Atif M. Memon, Mary Lou Soffa, and Martha E. Pollack. 2001. Coverage criteria for GUI testing. In Proceedings of the 8th European Software Engineering Conference held jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering 2001, Vienna, Austria, September 10-14, 2001. 256–267.
[45]
Guozhu Meng, Yinxing Xue, Chandramohan Mahinthan, Annamalai Narayanan, Yang Liu, Jie Zhang, and Tieming Chen. 2016. Mystique: Evolving Android Malware for Auditing Anti-Malware Tools. In Proceedings of the 11th ACM on Asia Conference on Computer and Communications Security (ASIA CCS ’16). ACM, New York, NY, USA, 365–376.
[46]
Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing Combinatorics in GUI Testing of Android Applications. In Proceedings of the 38th International Conference on Software Engineering (ICSE ’16). ACM, New York, NY, USA, 559–570.
[47]
Nariman Mirzaei, Sam Malek, Corina S. Pasareanu, Naeem Esfahani, and Riyadh Mahmood. 2012. Testing Android apps through symbolic execution. ACM SIGSOFT Software Engineering Notes 37, 6 (2012), 1–5.
[48]
Kevin Moran, Mario Linares Vásquez, Carlos Bernal-Cárdenas, Christopher Vendome, and Denys Poshyvanyk. 2016. Automatically Discovering, Reporting and Reproducing Android Application Crashes. In 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, Chicago, IL, USA, April 11-15, 2016. 33–44.
[49]
Cu D. Nguyen, Alessandro Marchetto, and Paolo Tonella. 2012. Combining modelbased and combinatorial testing for effective test case generation. In International Symposium on Software Testing and Analysis, ISSTA 2012, Minneapolis, MN, USA, July 15-20, 2012. 100–110.
[50]
Borislav Nikolik. 2006. Test diversity. Information & Software Technology 48, 11 (2006), 1083–1094.
[51]
Michael Pradel, Parker Schuh, George C. Necula, and Koushik Sen. 2014. Event-Break: analyzing the responsiveness of user interfaces through performanceguided test generation. In Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2014, part of SPLASH 2014, Portland, OR, USA, October 20-24, 2014. 33–47.
[52]
Stacy J. Prowell. 2005. Using Markov Chain Usage Models to Test Complex Systems. In 38th Hawaii International Conference on System Sciences (HICSS-38 2005), CD-ROM / Abstracts Proceedings, 3-6 January 2005, Big Island, HI, USA.
[53]
Vlad Roubtsov. 2017. EMMA. (2017). Retrieved 2017-2-18 from http://emma. sourceforge.net/
[54]
Eric Schkufza, Rahul Sharma, and Alex Aiken. 2013. Stochastic superoptimization. In Architectural Support for Programming Languages and Operating Systems, ASPLOS ’13, Houston, TX, USA - March 16 - 20, 2013. 305–316.
[55]
Muhammad Shafique and Yvan Labiche. 2010. A systematic review of model based testing tool support. Carleton University, Canada, Tech. Rep. Technical Report SCE-10-04 (2010).
[56]
Ting Su. 2016. FSMdroid: Guided GUI Testing of Android Apps. In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016 - Companion Volume. 689–691.
[57]
Ting Su, Ke Wu, Weikai Miao, Geguang Pu, Jifeng He, Yuting Chen, and Zhendong Su. 2017. A Survey on Data-Flow Testing. ACM Comput. Surv. 50, 1, Article 5 (March 2017), 35 pages.
[58]
Tommi Takala, Mika Katara, and Julian Harty. 2011. Experiences of System-Level Model-Based GUI Testing of an Android Application. In Fourth IEEE International Conference on Software Testing, Verification and Validation, ICST 2011, Berlin, Germany, March 21-25, 2011. 377–386.
[59]
Androguard Team. 2017. Androguard. (2017). Retrieved 2017-2-18 from https: //github.com/androguard/androguard
[60]
TextEdit Developers. 2017. TextEdit. (2017). Retrieved 2017-2-18 from https: //github.com/paulmach/Text-Edit-for-Android
[61]
Heila van der Merwe, Brink van der Merwe, and Willem Visser. 2012. Verifying Android Applications Using Java PathFinder. SIGSOFT Softw. Eng. Notes 37, 6 (Nov. 2012), 1–5.
[62]
Mario Linares Vásquez, Martin White, Carlos Bernal-Cárdenas, Kevin Moran, and Denys Poshyvanyk. 2015. Mining Android App Usages for Generating Actionable GUI-Based Execution Scenarios. In 12th IEEE/ACM Working Conference on Mining Software Repositories, MSR 2015, Florence, Italy, May 16-17, 2015. 111–122.
[63]
James A. Whittaker and Michael G. Thomason. 1994. A Markov Chain Model for Statistical Software Testing. IEEE Trans. Software Eng. 20, 10 (1994), 812–824.
[64]
Wikipedia. 2017. Cosine similarity. (2017). Retrieved 2017-2-18 from https: //en.wikipedia.org/wiki/Cosine_similarity
[65]
Wikipedia. 2017. Gibbs Sampling. (2017). Retrieved 2017-2-18 from https: //en.wikipedia.org/wiki/Gibbs_sampling
[66]
Wikipedia. 2017. Metropolis-Hastings algorithm. (2017). Retrieved 2017-2-18 from https://en.wikipedia.org/wiki/Metropolis-Hastings_algorithm
[67]
Qing Xie and Atif M. Memon. 2006. Studying the Characteristics of a "Good" GUI Test Suite. In 17th International Symposium on Software Reliability Engineering (ISSRE 2006), 7-10 November 2006, Raleigh, North Carolina, USA. 159–168.
[68]
Wei Yang, Mukul R. Prasad, and Tao Xie. 2013. A Grey-Box Approach for Automated GUI-Model Generation of Mobile Applications. In Fundamental Approaches to Software Engineering - 16th International Conference, FASE 2013, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2013, Rome, Italy, March 16-24, 2013. Proceedings. 250–265.
[69]
Bo Zhou, Hiroyuki Okamura, and Tadashi Dohi. 2010. Markov Chain Monte Carlo Random Testing. In Advances in Computer Science and Information Technology, AST/UCMA/ISA/ACN 2010 Conferences, Miyazaki, Japan, June 23-25, 2010. Joint Proceedings. 447–456.
[70]
Bo Zhou, Hiroyuki Okamura, and Tadashi Dohi. 2012. Application of Markov Chain Monte Carlo Random Testing to Test Case Prioritization in Regression Testing. IEICE Transactions 95-D, 9 (2012), 2219–2226.
[71]
Bo Zhou, Hiroyuki Okamura, and Tadashi Dohi. 2013. Enhancing Performance of Random Testing through Markov Chain Monte Carlo Methods. IEEE Trans. Computers 62, 1 (2013), 186–192.

Cited By

View all
  • (2025)CamDroid: Context-Aware Model-Based Automated GUI Testing for Android AppsTsinghua Science and Technology10.26599/TST.2024.901003830:1(55-67)Online publication date: Feb-2025
  • (2025)InforTest: Informer-Based Testing for Applications in the Internet of Robotic ThingsIEEE Transactions on Industrial Informatics10.1109/TII.2024.348570721:2(1499-1507)Online publication date: Feb-2025
  • (2025)Extraction and empirical evaluation of GUI-level invariants as GUI Oracles in mobile app testingInformation and Software Technology10.1016/j.infsof.2024.107531177:COnline publication date: 1-Jan-2025
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ESEC/FSE 2017: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering
August 2017
1073 pages
ISBN:9781450351058
DOI:10.1145/3106237
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. GUI Testing
  2. Mobile Apps
  3. Model-based Testing

Qualifiers

  • Research-article

Conference

ESEC/FSE'17
Sponsor:

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)205
  • Downloads (Last 6 weeks)11
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)CamDroid: Context-Aware Model-Based Automated GUI Testing for Android AppsTsinghua Science and Technology10.26599/TST.2024.901003830:1(55-67)Online publication date: Feb-2025
  • (2025)InforTest: Informer-Based Testing for Applications in the Internet of Robotic ThingsIEEE Transactions on Industrial Informatics10.1109/TII.2024.348570721:2(1499-1507)Online publication date: Feb-2025
  • (2025)Extraction and empirical evaluation of GUI-level invariants as GUI Oracles in mobile app testingInformation and Software Technology10.1016/j.infsof.2024.107531177:COnline publication date: 1-Jan-2025
  • (2025)ReckDroidComputers and Security10.1016/j.cose.2024.104117148:COnline publication date: 1-Jan-2025
  • (2024)UISGPT: Automated Mobile UI Design Smell Detection with Large Language ModelsElectronics10.3390/electronics1316312713:16(3127)Online publication date: 7-Aug-2024
  • (2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
  • (2024)A Model-Based Approach to Mobile Application TestingInternational Journal of Advanced Network, Monitoring and Controls10.2478/ijanmc-2023-00718:4(1-10)Online publication date: 16-Mar-2024
  • (2024)Navigating Mobile Testing Evaluation: A Comprehensive Statistical Analysis of Android GUI Testing MetricsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695476(944-956)Online publication date: 27-Oct-2024
  • (2024)Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChatProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695260(1973-1978)Online publication date: 27-Oct-2024
  • (2024)General and Practical Property-based Testing for Android AppsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694986(53-64)Online publication date: 27-Oct-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media