Mariana Souza
ABSTRACT
While the literature acknowledges that mobile apps present different testing challenges and automated solutions have been pursued, it lacks a better understanding of how pervasive practices of manual testing (namely Exploratory Testing - ET) can be more effectively applied. This paper aims to investigate the use of ET in mobile apps. With this study, we intend to have a better understanding of how exploratory testing is employed, its effectiveness, and its usage in an ample and diverse range of apps. To do so, we conducted two studies. The first study was conducted for the purpose of applying ET to apps with diverse contexts and available on Google Play in order to analyze whether testers actually explore all possible scenarios that apps may display. The second study, also applied the ET, however in two apps that were developed by a software development company; this study has the objective of applying the ET in order to identify bugs of different levels, that often cannot be revealed using other techniques. As expected the first study revealed that there are several test scenarios that are not exploited by the testers, yet the 40 participants revealed on average 5 bugs in 1.5h of test sessions. The second study revealed 64 bugs and 21 issues in two apps. Such revealed bugs are of different criticality and category. ET has shown to be a promising technique to uncover bugs, though test professionals can be better guided to explore their apps and search for bugs in scenarios related to mobile specific events.
- App Quality Alliance. 2018. AQuA Performance Testing Criteria. Retrieved from App Quality Alliance: http://www.appqualityalliance.org/aqua-performance-test-criteria (2018).Google Scholar
- Domenico Amalfitano, Vincenzo Riccio, Ana CR Paiva, and Anna Rita Fasolino. 2018. Why does the orientation change mess up my Android application? From GUI failures to code faults. Software Testing, Verification and Reliability 28, 1 (2018), e1654.Google ScholarCross Ref
- James Bach. 2003. Exploratory testing explained.Google Scholar
- Jerry Gao, Xiaoying Bai, Wei-Tek Tsai, and Tadahiro Uehara. 2014. Mobile Application Testing: A Tutorial. Computer 47, 2 (2014), 46--55. Google ScholarDigital Library
- Ceren \cSahin Gebizli and Hasan Sözer. 2017. Automated refinement of models for model-based testing using exploratory testing. Software Quality Journal 25, 3 (2017), 979--1005. Google ScholarDigital Library
- Hyung Kil Ham and Young Bom Park. 2011. Mobile application compatibility test system design for Android fragmentation. Communications in Computer and Information Science 257 CCIS (2011), 314--320.Google Scholar
- Juha Itkonen et al. 2011. Empirical studies on exploratory software testing. (2011).Google Scholar
- Juha Itkonen, Mika V Mantyla, and Casper Lassenius. 2007. Defect detection efficiency: Test case based vs. exploratory testing. In First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007). IEEE, 61--70. Google ScholarDigital Library
- Juha Itkonen and Kristian Rautiainen. 2005. Exploratory testing: a multiple case study. In 2005 International Symposium on Empirical Software Engineering, 2005. IEEE, 10--pp.Google ScholarCross Ref
- K. Rautiainen J. Itkonen. 2005. Exploratory testing: a multiple case study. In International Symposium on Empirical Software Engineering. 84--93.Google ScholarCross Ref
- Mona Erfani Joorabchi, Ali Mesbah, and Philippe Kruchten. 2013. Real challenges in mobile app development. In Empirical Software Engineering and Measurement, 2013 ACM/IEEE International Symposium on. IEEE, 15--24.Google ScholarCross Ref
- Cem Kaner, James Bach, and Bret Pettichord. 2008. Lessons learned in software testing. John Wiley & Sons.Google Scholar
- Cem Kaner, Jack Falk, and Hung Quoc Nguyen. 1999. Testing Computer Software.Google Scholar
- P. S. Kochhar, F. Thung, N. Nagappan, T. Zimmermann, and D. Lo. 2015. Understanding the Test Automation Culture of App Developers. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). 1--10.Google Scholar
- Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Kevin Moran, and Denys Poshyvanyk. 2017. How do developers test android applications?. In Proceedings - 2017 IEEE International Conference on Software Maintenance and Evolution, ICSME 2017. 613--622.Google ScholarCross Ref
- James Lyndsay and Neil Van Eeden. 2003. Adventures in session-based testing. Workroom Productions Ltd. May 27 (2003).Google Scholar
- Bakhtiar M. Amen, Sardasht M. Mahmood, and Joan Lu. 2015. Mobile Application Testing Matrix and Challenges. In Computer Science & Information Technology (CS & IT). 27--40.Google Scholar
- Henry Muccini, Antonio Di Francesco, and Patrizio Esposito. 2012. Software testing of mobile applications: Challenges and future research directions. In Proceedings of the 7th International Workshop on Automation of Software Test. IEEE Press, 29--35. Google ScholarDigital Library
- Glenford J Myers, Tom Badgett, Todd M Thomas, and Corey Sandler. 2004. The art of software testing. Vol. 2. Wiley Online Library. Google ScholarDigital Library
- Roger S Pressman. 2010. Software Engineering: A Practitioner's Approach, 7/e, RS Pressman & Associates. Inc., McGraw-Hill, ISBN 73375977 (2010). Google ScholarDigital Library
- Rudolf Ramler, Georg Buchgeher, and Claus Klammer. 2018. Adapting automated test generation to GUI testing of industry applications. Information and Software Technology 93 (2018), 248--263. Google ScholarDigital Library
- Reinaldo Costa Santana. 2008. Computação móvel, histórico da evolução. Acesso em 11 (2008).Google Scholar
- Davi Bernardo Silva, Marcelo Medeiros Eler, Vinicius HS Durelli, and Andre Takeshi Endo. 2018. Characterizing mobile apps from a source and test code viewpoint. Information and Software Technology 101 (2018), 32--50.Google ScholarCross Ref
- Statista. 2019. Number of apps available in leading app stores as of 1st quarter 2019. https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/Google Scholar
- Andy Tinkham and Cem Kaner. 2003. Learning styles and exploratory testing. In Proceedings of the Pacific northwest software quality conference. 00--00.Google Scholar
- Anthony I Wasserman and Fosser. 2010. Software Engineering Issues for Mobile Application Development. In FoSER '10 Proceedings of the FSE/SDP workshop on Future of software engineering research. 397--400. Google ScholarDigital Library
- James A Whittaker. 2009. Exploratory software testing: tips, tricks, tours, and techniques to guide test design. Pearson Education. Google ScholarDigital Library
- James A. Whittaker. 2009. Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design (1st ed.). Addison-Wesley Professional. Google ScholarDigital Library
- Samer Zein, Norsaremah Salleh, and John Grundy. 2016. A systematic mapping study of mobile application testing techniques. Journal of Systems and Software 117 (2016), 334--356. Google ScholarDigital Library
Index Terms
- On the Exploratory Testing of Mobile Apps
Recommendations
Exploratory testing of apps with opportunity maps
SBQS '21: Proceedings of the XX Brazilian Symposium on Software QualityExploratory Testing (ET) is a well-known technique to perform manual tests. Its main goal is to foster creativity and freedom, while the tester learns, designs and executes tests continuously in a time-boxed session. Popular among agile teams, ET is ...
Mutation operators for testing Android apps
Context: Due to the widespread use of Android devices, Android applications (apps) have more releases, purchases, and downloads than apps for any other mobile devices. The sheer volume of code in these apps creates significant concerns about the quality ...
Reuse of model-based tests in mobile apps
SBES '17: Proceedings of the XXXI Brazilian Symposium on Software EngineeringMobile apps have been introduced in our lives and as a specific class of software, developers and testers have to deal with new challenges. For instance, testing all configurations and characteristics of apps might be an expensive activity. It would be ...
Comments