Abstract
We consider test design as an optimization problem. The challenge is to find a set of test cases, the so-called test suite, that optimizes two quantifiable requirements: First, the effort needed for test execution should be minimal; a given test budget usually sets a maximum for the size of the test suite. Second, the test suite should maximize the score of a given test goal estimating its error detection capability, e.g., by the use of coverage or risk metrics. This paper studies test design for testing self-organizing systems with a mutation-based test goal. Equipped with a reconfiguration mechanism, this kind of a distributed system adapts its internal structure and thus its behavior to changing environmental conditions at run time. Test execution at a time step t consequently not only triggers an observable output at \(t+1\), but might also bring about a reconfiguration of the system under test influencing the result of subsequently executed test cases. Formalizing the evolving sequential decision problem of test case executions by dependency graphs, in which we try to find optimal sets of paths for the mutation-based goal, we investigate the suitability of various kinds of evolutionary algorithms for optimization. All of the considered algorithms are evaluated using a concrete case study of an adaptive, self-organizing production cell.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D.: Concrete problems in AI safety (2016). https://arxiv.org/pdf/1606.06565
Arjovsky, M., Bottou, L.: Towards principled methods for training generative adversarial networks (2017). https://arxiv.org/pdf/1701.04862
Belzner, L., Beck, M.T., Gabor, T., Roelle, H., Sauer, H.: Software engineering for distributed autonomous real-time systems. In: Proceedings of the 2nd International Workshop on Software Engineering for Smart Cyber-Physical Systems, pp. 54–57. ACM (2016)
Belzner, L., Gabor, T.: Bayesian verification under model uncertainty. In: Proceedings of the 3rd International Workshop on Software Engineering for Smart Cyber-Physical Systems, pp. 10–13. IEEE (2017)
Black, J., Melachrinoudis, E., Kaeli, D.: Bi-criteria models for all-uses test suite reduction. In: Proceedings of the 26th International Conference on Software Engineering, pp. 106–115. IEEE (2004)
Bures, T., Weyns, D., Klein, M., Haber, R.E.: 1st International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS 2015). In: Proceedings of the 37th International Conference on Software Engineering, vol. 2, pp. 1009–1010. IEEE (2015)
Calinescu, R., Ghezzi, C., Kwiatkowska, M., Mirandola, R.: Self-adaptive software needs quantitative verification at runtime. Comm. ACM 55(9), 69–77 (2012)
de Lemos, R., et al.: Software engineering for self-adaptive systems: a second research roadmap. In: de Lemos, R., Giese, H., Müller, H.A., Shaw, M. (eds.) Software Engineering for Self-Adaptive Systems II. LNCS, vol. 7475, pp. 1–32. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-35813-5_1
DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. Computer 11(4), 34–41 (1978)
Eberhardinger, B., Seebach, H., Klumpp, D., Reif, W.: Test case selection strategy for self-organization mechanisms. In: Spillner, A., Winter, M., Pietschker, A. (eds.) Test, Analyse und Verifikation von Software – Gestern, Heute, Morgen, pp. 139–157. dpunkt (2017)
Eberhardinger, B., Seebach, H., Knapp, A., Reif, W.: Towards testing self-organizing, adaptive systems. In: Merayo, M.G., de Oca, E.M. (eds.) ICTSS 2014. LNCS, vol. 8763, pp. 180–185. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44857-1_13
Eiben, A.E., Smith, J.E.: Introduction to Evolutionary Computing. Natural Computing Series, vol. 53. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-662-05094-1
Fraser, G., Arcuri, A.: Evosuite: automatic test suite generation for object-oriented software. In: Proceedings of the 19th ACM SIGSOFT Symposium and 13th European Conference on Foundations of Software Engineering, pp. 416–419. ACM (2011)
Fredericks, E.M., Ramirez, A.J., Cheng, B.H.C.: Towards run-time testing of dynamic adaptive systems. In: Proceedings of the 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, pp. 169–174 (2013)
Gabor, T., Belzner, L., Linnhoff-Popien, C.: Inheritance-based diversity measures for explicit convergence control in evolutionary algorithms. In: Proceedings of the Genetic and Evolutionary Computation Conference (2018)
Grefenstette, J.J.: Genetic algorithms for changing environments. In: Proceedings of the Parallel Problem Solving from Nature, vol. 2, pp. 139–146. Elsevier (1992)
Güdemann, M., Nafz, F., Ortmeier, F., Seebach, H., Reif, W.: A specification and construction paradigm for organic computing systems. In: Proceedings of the 2nd International Conference on Self-adaptive and Self-organizing Systems, pp. 233–242. IEEE (2008)
Ishibuchi, H., Shibata, Y.: A similarity-based mating scheme for evolutionary multiobjective optimization. In: Cantú-Paz, E. (ed.) GECCO 2003, Part I. LNCS, vol. 2723, pp. 1065–1076. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-45105-6_116
Lowd, D., Meek, C.: Adversarial learning. In: Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, pp. 641–647. ACM (2005)
McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab. 14(2), 105–156 (2004)
Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing. Wiley, Hoboken (2011)
Reichstaller, A., Eberhardinger, B., Knapp, A., Reif, W., Gehlen, M.: Risk-based interoperability testing using reinforcement learning. In: Wotawa, F., Nica, M., Kushik, N. (eds.) ICTSS 2016. LNCS, vol. 9976, pp. 52–69. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47443-4_4
Reichstaller, A., Eberhardinger, B., Ponsar, H., Knapp, A., Reif, W.: Test suite reduction for self-organizing systems: a mutation-based approach. In: Proceedings of the 13th International Workshop on Automation of Software Test (2018)
Rothermel, G., Harrold, M.J., Von Ronne, J., Hong, C.: Empirical studies of test-suite reduction. Softw. Test. Verif. Reliab. 12(4), 219–249 (2002)
Ursem, R.K.: Diversity-guided evolutionary algorithms. In: Guervós, J.J.M., Adamidis, P., Beyer, H.-G., Schwefel, H.-P., Fernández-Villacañas, J.-L. (eds.) PPSN 2002. LNCS, vol. 2439, pp. 462–471. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45712-7_45
Wappler, S., Lammermann, F.: Using evolutionary algorithms for the unit testing of object-oriented software. In: Proceedings of the Annual Conference on Genetic and Evolutionary Computation, pp. 1053–1060. ACM (2005)
Weicker, K.: Evolutionary Algorithms and Dynamic Optimization Problems. Der Andere Verlag, Berlin (2003)
Wirsing, M., Hölzl, M., Koch, N., Mayer, P. (eds.): Software Engineering for Collective Autonomic Systems: The ASCENS Approach. LNCS, vol. 8998. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16310-9
Zhang, L., Marinov, D., Zhang, L., Khurshid, S.: An empirical study of JUnit test-suite reduction. In: Proceedings of the 22nd International Symposium on Software Reliability Engineering, pp. 170–179. IEEE (2011)
Acknowledgment
This research is partly funded by the research project Testing self-organizing, adaptive Systems (TeSOS) of the German Research Foundation. We thank the anonymous reviewers for their helpful comments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Reichstaller, A., Gabor, T., Knapp, A. (2018). Mutation-Based Test Suite Evolution for Self-Organizing Systems. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation. Distributed Systems. ISoLA 2018. Lecture Notes in Computer Science(), vol 11246. Springer, Cham. https://doi.org/10.1007/978-3-030-03424-5_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-03424-5_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03423-8
Online ISBN: 978-3-030-03424-5
eBook Packages: Computer ScienceComputer Science (R0)