Abstract
Search-based approaches have become a standard technique in automated Android testing. However, deceptive fitness landscapes make it difficult to find optimal solutions in a reasonable time. An alternative to classical search algorithms is offered by novelty search algorithms, which replace classical coverage-based fitness functions with a fitness function that rewards chromosomes based on how different, i.e., novel, they are compared to prior solutions. An open question is how to best quantify novelty for Android tests. In this paper, we therefore study two alternative approaches: First, we use a k-nearest neighbours (kNN) approach to determine the novelty in terms of coverage; second, we quantify the novelty of a chromosome based on the second-order uncertainty derived from subjective logic. We integrated both approaches into the open source Android test generator MATE and study its performance on 108 complex apps. Our experiments suggest that novelty search represents a promising alternative but requires further research to fully unleash its power: While both novelty search variants outperform a random-based exploration, they cannot yet compete with the state-of-the-art many-objective search algorithm MIO.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Amalfitano, D., Amatucci, N., Fasolino, A.R., Tramontana, P.: AGRippin: a novel search based testing technique for android applications. In: Proceedings of DeMobile, pp. 5–12. ACM (2015)
Arcuri, A.: Many independent objective (MIO) algorithm for test suite generation. In: Menzies, T., Petke, J. (eds.) SSBSE 2017. LNCS, vol. 10452, pp. 3–17. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66299-2_1
Auer, M., Adler, F., Fraser, G.: Improving search-based android test generation using surrogate models. In: Papadakis, M., Vergilio, S.R. (eds.) SSBSE 2022. LNCS, vol. 13711, pp. 51–66. Springer, Cham (2022)
Boussaa, M., Barais, O., Sunyé, G., Baudry, B.: A novelty search approach for automatic test data generation. In: Procedings of SBST, pp. 40–43 (2015)
Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-ii. IEEE TEC 6(2), 182–197 (2002)
DeVries, B., Trefftz, C.: A novelty search and metamorphic testing approach to automatic test generation. In: Proceedings of SBST, pp. 8–11. IEEE (2021)
Doncieux, S., Laflaquière, A., Coninx, A.: Novelty search: A theoretical perspective. In: Proceedings of GECCO, p. 99–106. ACM (2019)
Eiben, A.E., Smith, J.E.: Introduction to Evolutionary Computing, 2nd edn. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-44874-8
Eler, M.M., Rojas, J.M., Ge, Y., Fraser, G.: Automated accessibility testing of mobile apps. In: ICST, pp. 116–126. IEEE (2018)
Gomes, J., Mariano, P., Christensen, A.L.: Devising effective novelty search algorithms: a comprehensive empirical study. In: Proceedings of GECCO, pp. 943–950. ACM (2015)
Gomes, J., Urbano, P., Christensen, A.L.: Progressive minimal criteria novelty search. In: Pavón, J., Duque-Méndez, N.D., Fuentes-Fernández, R. (eds.) IBERAMIA 2012. LNCS (LNAI), vol. 7637, pp. 281–290. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-34654-5_29
Jøsang, A.: Subjective Logic: A Formalism for Reasoning Under Uncertainty, 1st edn. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-42337-1
Kong, P., Li, L., Gao, J., Liu, K., Bissyandé, T.F., Klein, J.: Automated testing of android apps: a systematic literature review. IEEE Trans. Reliab. 68(1), 45–66 (2019)
Lehman, J., Stanley, K.O.: Exploiting open-endedness to solve problems through the search for novelty. In: IEEE Symposium on Artificial Life (2008)
Liapis, A., Yannakakis, G.N., Togelius, J.: Enhancements to constrained novelty search: two-population novelty search for generating game content. In: Proceedings of GECCO, pp. 343–350. ACM (2013)
Mahmood, R., Mirzaei, N., Malek, S.: Evodroid: segmented evolutionary testing of android apps. In: Proceedings of FSE, pp. 599–609. ACM (2014)
Mao, K., Harman, M., Jia, Y.: Sapienz: multi-objective automated testing for android applications. In: Proceedings of ISSTA, pp. 94–105. ACM (2016)
Mouret, J.B., Doncieux, S.: Encouraging behavioral diversity in evolutionary robotics: an empirical study. Evol. Comput. 20(1), 91–133 (2012)
Mouret, J.B.: Novelty-based multiobjectivization. In: Doncieux, S., Bredèche, N., Mouret, J.B. (eds.) New Horizons in Evolutionary Robotics. Studies in Computational Intelligence, vol. 341, pp. 139–154. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-18272-3_10
Naredo, E., Trujillo, L.: Searching for novel clustering programs. In: Proceedings of GECCO, pp. 1093–1100. ACM (2013)
Romdhana, A., Merlo, A., Ceccato, M., Tonella, P.: Deep reinforcement learning for black-box testing of android apps. TOSEM 31, 1–29 (2022)
Rummery, G., Niranjan, M.: On-line q-learning using connectionist systems. Technical report CUED/F-INFENG/TR 166 (1994)
Salehi, A., Coninx, A., Doncieux, S.: BR-NS: an archive-less approach to novelty search. In: Proceedings of GECCO, pp. 172–179. ACM (2021)
Sell, L., Auer, M., Frädrich, C., Gruber, M., Werli, P., Fraser, G.: An empirical evaluation of search algorithms for app testing. In: Gaston, C., Kosmatov, N., Le Gall, P. (eds.) ICTSS 2019. LNCS, vol. 11812, pp. 123–139. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31280-0_8
Shi, L., Li, S., Zheng, Q., Yao, M., Pan, G.: Efficient novelty search through deep reinforcement learning. IEEE Access 8, 128809–128818 (2020)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. A Bradford Book (2018)
Vargha, A., Delaney, H.D.: A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J. Educ. Behav. Stat. 25(2), 101–132 (2000)
Villanueva, O.M., Trujillo, L., Hernandez, D.E.: Novelty search for automatic bug repair. In: Proceedings of GECCO, pp. 1021–1028. ACM (2020)
Walkinshaw, N., Hierons, R.M.: Modelling second-order uncertainty in state machines. IEEE TSE 49(5), 3261–3276 (2023)
Acknowledgment
This work is supported by DFG project FR2955/4-1 “STUNT: Improving Software Testing Using Novelty”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Auer, M., Pusl, M., Fraser, G. (2024). Generating Android Tests Using Novelty Search. In: Arcaini, P., Yue, T., Fredericks, E.M. (eds) Search-Based Software Engineering. SSBSE 2023. Lecture Notes in Computer Science, vol 14415. Springer, Cham. https://doi.org/10.1007/978-3-031-48796-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-48796-5_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-48795-8
Online ISBN: 978-3-031-48796-5
eBook Packages: Computer ScienceComputer Science (R0)