Stefan Fischer
Alexander Egyed
ABSTRACT
Testing is a fundamental verification activity to produce high-quality software. However, testing is a costly and complex activity. The success of software testing depends on the quality of test cases but finding a good set of test cases is laborious. To make matters worse, when dealing with a family of systems (e.g., variants of a mobile applications), test cases must assure that a diversity of configurations in potentially many variants work as expected. This is the case of hello again GmbH, a company that develops mobile applications for customer loyalty (e.g., discounts, free products, rewards, or insider perks). The company targets several business domains, and currently supports about 700 application variants. Testing such applications including all their variability is a cumbersome task. Even simple test cases designed for one variant most likely cannot be reused for other variants. To support developers at hello again GmbH, we present a solution to employ a model-based testing approach to their family of mobile apps. Model-based testing focuses on automatizing the design and generation of test cases. We present results of applying model-based testing on 27 applications from hello again GmbH and report the challenges and lessons learned for designing a variable test model. Our expected contribution is to support companies and practitioners looking for solutions to test families of software products.
- Tanwir Ahmad, Junaid Iqbal, Adnan Ashraf, Dragos Truscan, and Ivan Porres. 2019. Model-based testing using UML activity diagrams: A systematic mapping study. Computer Science Review 33 (2019), 98--112.Google Scholar
Digital Library
- Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M. Memon. 2012. Using GUI ripping for automated testing of Android applications. In IEEE/ACM International Conference on Automated Software Engineering, ASE' 12, Essen, Germany, September 3-7, 2012. ACM, 258--261. https://doi.org/10.1145/2351676.2351717Google ScholarDigital Library
- Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Bryan Dzung Ta, and Atif M. Memon. 2015. MobiGUITAR: Automated Model-Based Testing of Mobile Apps. IEEE Softw. 32, 5 (2015), 53--59. https://doi.org/10.1109/MS.2014.55Google ScholarDigital Library
- Aitor Arrieta, Goiuria Sagardui, and Leire Etxeberria. 2014. A model-based testing methodology for the systematic validation of highly configurable cyber-physical systems. In 6th International Conference on Advances in System Testing and Validation Lifecycle. IARIA XPS Press, 66--72.Google Scholar
- Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2013, part of SPLASH 2013, Indianapolis, IN, USA, October 26-31, 2013. ACM, 641--660. https://doi.org/10.1145/2509136.2509549Google ScholarDigital Library
- Antonia Bertolino. 2007. Software testing research: Achievements, challenges, dreams. In Future of Software Engineering (FOSE'07). IEEE, 85--103.Google Scholar
- J. Bosch, R. Capilla, and R. Hilliard. 2015. Trends in Systems and Software Variability. IEEE Software 32, 3 (2015), 44--51.Google ScholarDigital Library
- Rafael Capilla, J. Bosch, and K. C. Kang. 2013. Systems and Software Variability Management: Concepts, Tools and Experiences. Springer.Google Scholar
- Wontae Choi, George C. Necula, and Koushik Sen. 2013. Guided GUI testing of android apps with minimal restart and approximate learning. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2013, part of SPLASH 2013, Indianapolis, IN, USA, October 26-31, 2013. ACM, 623--640. https://doi.org/10.1145/2509136.2509552Google ScholarDigital Library
- Andreas Classen, Maxime Cordy, Pierre-Yves Schobbens, Patrick Heymans, Axel Legay, and Jean-Francois Raskin. 2013. Featured Transition Systems: Foundations for Verifying Variability-Intensive Systems and Their Application to LTL Model Checking. IEEE Transactions on Software Engineering 39, 8 (2013), 1069--1089. https://doi.org/10.1109/TSE.2012.86Google ScholarDigital Library
- Andreas Classen, Patrick Heymans, Pierre-Yves Schobbens, Axel Legay, and Jean-Francois Raskin. 2010. Model checking lots of systems: efficient verification of temporal properties in software product lines. In 2010 ACM/IEEE 32nd International Conference on Software Engineering, Vol. 1. 335--344. https://doi.org/10.1145/1806799.1806850Google ScholarDigital Library
- S.R. Dalal, A. Jain, N. Karunanithi, J.M. Leaton, C.M. Lott, G.C. Patton, and B.M. Horowitz. 1999. Model-based testing in practice. In International Conference on Software Engineering. 285--294. https://doi.org/10.1145/302405.302640Google ScholarDigital Library
- Edsger W Dijkstra. 2022. A note on two problems in connexion with graphs. In Edsger Wybe Dijkstra: His Life, Work, and Legacy. 287--290.Google Scholar
- Ivan do Carmo Machado. 2014. Fault model-based variability testing. Ph.D. Dissertation. Federal University of Bahia, Salvador, Brazil. http://repositorio.ufba.br/ri/handle/ri/22826Google Scholar
- Christian Dziobek and Jens Weiland. 2009. Variantenmodellierung und -konfiguration eingebetteter automotive Software mit Simulink. In Dagstuhl-Workshop MBEES: Modellbasierte Entwicklung eingebetteter Systeme V, Schloss Dagstuhl, Germany, 2009, Tagungsband Modellbasierte Entwicklung eingebetteter Systeme (Informatik-Bericht, Vol. 2009-01). TU Braunschweig, Institut für Software Systems Engineering, 36--45. http://www.sse-tubs.de/mbees-dagstuhl/MBEES2009_Proceedings_online_small.pdfGoogle Scholar
- Stefan Fischer, Gabriela Karoline Michelon, Wesley K. G. Assunção, Rudolf Ramler, and Alexander Egyed. 2023. Designing a Test Model for a Configurable System: An Exploratory Study of Preprocessor Directives and Feature Toggles. In 17th International Working Conference on Variability Modelling of Software-Intensive Systems (VaMoS). ACM, 31--39. https://doi.org/10.1145/3571788.3571795Google ScholarDigital Library
- Stefan Fischer, Gabriela Karoline Michelon, Rudolf Ramler, Lukas Linsbauer, and Alexander Egyed. 2020. Automated test reuse for highly configurable software. Empir. Softw. Eng. 25, 6 (2020), 5295--5332. https://doi.org/10.1007/s10664-020-09884-xGoogle ScholarDigital Library
- Stefan Fischer, Rudolf Ramler, and Lukas Linsbauer. 2021. Comparing Automated Reuse of Scripted Tests and Model-Based Tests for Configurable Software. In 28th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 421--430. https://doi.org/10.1109/APSEC53868.2021.00049Google Scholar
- Stefan Fischer, Rudolf Ramler, Lukas Linsbauer, and Alexander Egyed. 2019. Automating test reuse for highly configurable software. In Proceedings of the 23rd International Systems and Software Product Line Conference, SPLC 2019, Volume A, Paris, France, September 9-13, 2019. ACM, 1:1-1:11. https://doi.org/10.1145/3336294.3336305Google ScholarDigital Library
- Vahid Garousi and Junji Zhi. 2013. A survey of software testing practices in Canada. Journal of Systems and Software 86, 5 (2013), 1354--1376.Google ScholarDigital Library
- Lorenzo Gomez, Iulian Neamtiu, Tanzirul Azim, and Todd D. Millstein. 2013. RERAN: timing- and touch-sensitive record and replay for Android. In 35th International Conference on Software Engineering, ICSE '13, San Francisco, CA, USA, May 18-26, 2013. IEEE Computer Society, 72--81. https://doi.org/10.1109/ICSE.2013.6606553Google Scholar
- Hans Grönniger, Holger Krahn, Claas Pinkernell, and Bernhard Rumpe. 2014. Modeling Variants of Automotive Systems using Views. CoRR abs/1409.6629 (2014). arXiv:1409.6629 http://arxiv.org/abs/1409.6629Google Scholar
- Havva Gulay Gurbuz and Bedir Tekinerdogan. 2018. Model-based testing for software safety: a systematic mapping study. Software Quality Journal 26, 4 (2018), 1327--1372.Google ScholarDigital Library
- Günter Halmans and Klaus Pohl. 2003. Communicating the Variability of a Software-Product Family to Customers. Software and System Modeling 2, 1 (2003), 15--36.Google ScholarCross Ref
- Yongjian Hu, Tanzirul Azim, and Iulian Neamtiu. 2015. Versatile yet lightweight record-and-replay for Android. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SPLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015. ACM, 349--366. https://doi.org/10.1145/2814270.2814320Google ScholarDigital Library
- Kyo Kang, Sholom Cohen, James Hess, William Novak, and A. Peterson. 1990. Feature-Oriented Domain Analysis (FODA) Feasibility Study. Technical Report CMU/SEI-90-TR-021. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=11231Google Scholar
- Teemu Kanstrén and Olli-Pekka Puolitaival. 2012. Using Built-In Domain-Specific Modeling Support to Guide Model-Based Test Generation. In Proceedings 7th Workshop on Model-Based Testing, MBT 2012, Tallinn, Estonia, 25 March 2012 (EPTCS, Vol. 80). 58--72. https://doi.org/10.4204/EPTCS.80.5Google ScholarCross Ref
- Stefan Karlsson, Adnan Causevic, Daniel Sundmark, and Mårten Larsson. 2021. Model-based Automated Testing of Mobile Applications: An Industrial Case Study. In 14th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICST Workshops 2021, Porto de Galinhas, Brazil, April 12-16, 2021. IEEE, 130--137. https://doi.org/10.1109/ICSTW52544.2021.00033Google Scholar
- Stefan Kriebel, Matthias Markthaler, Karin Samira Salman, Timo Greifenberg, Steffen Hillemacher, Bernhard Rumpe, Christoph Schulze, Andreas Wortmann, Philipp Orth, and Johannes Richenhagen. 2018. Improving model-based testing in automotive software engineering. In 40th International Conference on Software Engineering: Software Engineering in Practice. 172--180.Google ScholarDigital Library
- Axel Legay, Gilles Perrouin, Xavier Devroey, Maxime Cordy, Pierre-Yves Schobbens, and Patrick Heymans. 2017. On Featured Transition Systems. In SOFSEM 2017: Theory and Practice of Computer Science. Springer International Publishing, Cham, 453--463.Google Scholar
- Maurizio Leotta, Matteo Biagiola, Filippo Ricca, Mariano Ceccato, and Paolo Tonella. 2020. A Family of Experiments to Assess the Impact of Page Object Pattern in Web Test Suite Development. In 13th IEEE International Conference on Software Testing, Validation and Verification, ICST 2020, Porto, Portugal, October 24-28, 2020. IEEE, 263--273. https://doi.org/10.1109/ICST46399.2020.00035Google ScholarCross Ref
- Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: an input generation system for Android apps. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE'13, Saint Petersburg, Russian Federation, August 18-26, 2013. ACM, 224--234. https://doi.org/10.1145/2491411.2491450Google ScholarDigital Library
- Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbrücken, Germany, July 18-20, 2016. ACM, 94--105. https://doi.org/10.1145/2931037.2931054Google ScholarDigital Library
- Sebastian Oster, Andreas Wübbeke, Gregor Engels, and Andy Schürr. 2011. A Survey of Model-Based Software Product Lines Testing. In Model-Based Testing for Embedded Systems. CRC Press. https://doi.org/10.1201/b11321-14Google Scholar
- Owain Parry, Gregory M Kapfhammer, Michael Hilton, and Phil McMinn. 2021. A survey of flaky tests. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021), 1--74.Google ScholarDigital Library
- Kleber L. Petry, Edson OliveiraJr, and Avelino F. Zorzo. 2020. Model-based testing of software product lines: Mapping study and research roadmap. J. Syst. Softw. 167 (2020), 110608. https://doi.org/10.1016/j.jss.2020.110608Google ScholarCross Ref
- Klaus Pohl, Günter Böckle, and Frank J. van der Linden. 2005. Software Product Line Engineering: Foundations, Principles and Techniques (1 ed.). Springer.Google ScholarDigital Library
- Alexander Pretschner. 2005. Model-Based Testing in Practice. In FM 2005: Formal Methods. Springer Berlin Heidelberg, Berlin, Heidelberg, 537--541.Google Scholar
- Andreas Reuys, Erik Kamsties, Klaus Pohl, and Sacha Reis. 2005. Model-Based System Testing of Software Product Families. In 17th International Conference Advanced Information Systems Engineering (CAiSE) (Lecture Notes in Computer Science, Vol. 3520). Springer, 519--534. https://doi.org/10.1007/11431855_36Google ScholarDigital Library
- Kabir S. Said, Liming Nie, Adekunle Akinjobi Ajibode, and Xueyi Zhou. 2020. GUI testing for mobile applications: objectives, approaches and challenges. In Internetware '20: 12th Asia-Pacific Symposium on Internetware, Singapore, November 1-3, 2020. ACM, 51--60. https://doi.org/10.1145/3457913.3457931Google ScholarDigital Library
- Klaus Schmid and Isabel John. 2004. A customizable approach to full lifecycle variability management. Science of Computer Programming 53, 3 (2004), 259--284.Google ScholarDigital Library
- Porfirio Tramontana, Domenico Amalfitano, Nicola Amatucci, and Anna Rita Fasolino. 2019. Automated functional testing of mobile applications: a systematic mapping study. Softw. Qual. J. 27, 1 (2019), 149--201. https://doi.org/10.1007/s11219-018-9418-6Google ScholarDigital Library
- Mark Utting and Bruno Legeard. 2010. Practical model-based testing: a tools approach. Elsevier.Google ScholarDigital Library
- Mark Utting, Bruno Legeard, Fabrice Bouquet, Elizabeta Fourneret, Fabien Peureux, and Alexandre Vernotte. 2016. Chapter Two - Recent Advances in Model-Based Testing. Advances in Computers, Vol. 101. Elsevier, 53--120. https://doi.org/10.1016/bs.adcom.2015.11.004Google Scholar
Index Terms
- Model-based Testing for a Family of Mobile Applications: Industrial Experiences
Recommendations
Reusable Model-Based Testing
ICSR '09: Proceedings of the 11th International Conference on Software Reuse: Formal Foundations of Reuse and Domain EngineeringA reusable model-based testing method for software product lines (SPL) is used to create test specifications from use case and feature models, which can then be configured to test individual applications that are members of the SPL. This paper describes ...
Model-based pairwise testing for feature interaction coverage in software product line engineering
Testing software product lines (SPLs) is very challenging due to a high degree of variability leading to an enormous number of possible products. The vast majority of today's testing approaches for SPLs validate products individually using different ...
Risk-based integration testing of software product lines
VaMoS '17: Proceedings of the 11th International Workshop on Variability Modelling of Software-Intensive SystemsSoftware product lines (SPL) capture commonalities and variabilities of product families and, thus, enable mass customization of product variants according to customers desired configurations. However, they introduce new challenges to software testing ...
Comments