André Reichstaller
ABSTRACT
We study regression testing and test suite reduction for self-organizing (SO) systems. The complex environments of SO systems typically require large test suites. The physical distribution of their components and their history-dependent behavior, however, make test execution very expensive. Consequently, an efficient test suite reduction mechanism is needed. The fundamental characteristic of SO systems is their ability to reconfigure themselves. We thus investigate a mutation-based approach concentrating on reconfigurations, more specifically the communication between the distributed components in reconfigurations. Due to distribution, we argue for an explicit consideration of higher-order mutants and find a short-cut that makes the number of test cases to execute before reduction feasible. For the reduction task, we evaluate the applicability of two existing clustering techniques, Affinity Propagation and Dissimilarity-based Sparse Subset Selection. It turns out that these techniques are able to drastically reduce the original test suite while retaining a good mutation score. We discuss the approach by means of a test suite for a self-organizing production cell as a running example.
- Laura Balzano, Robert Nowak, and Waheed Bajwa. 2010. Column subset selection with missing data. In NIPS workshop on low-rank methods for large-scale machine learning, Vol. 1. Citeseer.Google Scholar
- Jennifer Black, Emanuel Melachrinoudis, and David Kaeli. 2004. Bi-criteria models for all-uses test suite reduction. In Proceedings of the 26th International Conference on Software Engineering. IEEE Computer Society, 106--115. Google ScholarDigital Library
- Christos Boutsidis, Michael W Mahoney, and Petros Drineas. 2009. An improved approximation algorithm for the column subset selection problem. In Proceedings of the twentieth annual ACM-SIAM symposium on Discrete algorithms. Society for Industrial and Applied Mathematics, 968--977. Google ScholarDigital Library
- Rogério de Lemos et al. 2013. Software Engineering for Self-Adaptive Systems: A Second Research Roadmap. In Software Engineering for Self-Adaptive Systems II, Rogério de Lemos, Holger Giese, Hausi A. Müller, and Mary Shaw (Eds.). LNCS, Vol. 7475. Springer, 1--32.Google Scholar
- Richard A DeMillo, Richard J Lipton, and Frederick G Sayward. 1978. Hints on test data selection: Help for the practicing programmer. Computer 11, 4 (1978), 34--41. Google ScholarDigital Library
- Benedikt Eberhardinger, Gerrit Anders, Hella Seebach, Florian Siefert, Alexander Knapp, and Wolfgang Reif. 2017. An approach for isolated testing of self-organization algorithms. In Software Engineering for Self-Adaptive Systems III. Assurances. Springer, 188--222.Google Scholar
- Benedikt Eberhardinger, Axel Habermaier, Hella Seebach, and Wolfgang Reif. 2016. Back-to-back testing of self-organization mechanisms. In IFIP International Conference on Testing Software and Systems. Springer, 18--35.Google ScholarCross Ref
- Benedikt Eberhardinger, Hella Seebach, Dominik Klumpp, and Wolfgang Reif. 2017. Test Case Selection Strategy for Self-Organization Mechanisms. In Test, Analyse und Verifikation von Software - gestern, heute, morgen. dpunkt, 139--157.Google Scholar
- Ehsan Elhamifar, Guillermo Sapiro, and S Shankar Sastry. 2016. Dissimilarity-based sparse subset selection. IEEE transactions on pattern analysis and machine intelligence 38, 11 (2016), 2182--2197. Google ScholarDigital Library
- Ehsan Elhamifar, Guillermo Sapiro, and Rene Vidal. 2012. Finding exemplars from pairwise dissimilarities via simultaneous sparse recovery. In Advances in Neural Information Processing Systems. 19--27. Google ScholarDigital Library
- Stephanie Forrest. 1990. Emergent computation: Self-organizing, collective, and cooperative phenomena in natural and artificial computing networks. Physica D: Nonlinear Phenomena 42, 1 (1990), 1--11. Google ScholarDigital Library
- Gordon Fraser and Franz Wotawa. 2007. Redundancy based test-suite reduction. In International Conference on Fundamental Approaches to Software Engineering. Springer, 291--305. Google ScholarDigital Library
- Brendan J Frey and Delbert Dueck. 2007. Clustering by passing messages between data points. science 315, 5814 (2007), 972--976.Google Scholar
- Richard W Hamming. 1950. Error detecting and error correcting codes. Bell Labs Technical Journal 29, 2 (1950), 147--160.Google ScholarCross Ref
- Yue Jia and Mark Harman. 2008. Constructing subtle faults using higher order mutation testing. In Source Code Analysis and Manipulation, 2008 Eighth IEEE International Working Conference on. IEEE, 249--258.Google ScholarCross Ref
- James A Jones and Mary Jean Harrold. 2003. Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Transactions on Software Engineering 29, 3 (2003), 195--209. Google ScholarDigital Library
- Leonard Kaufman and Peter Rousseeuw. 1987. Clustering by means of medoids. North-Holland.Google Scholar
- A. Jefferson Offutt. 1989. The coupling effect: fact or fiction. In ACM SIGSOFT Software Engineering Notes, Vol. 14. ACM, 131--140. Google ScholarDigital Library
- A. Jefferson Offutt, Jie Pan, and Jeffrey M. Voas. 1995. Procedures for reducing the size of coverage-based test sets. In Proceedings of International Conference on Testing Computer Software.Google Scholar
- André Reichstaller, Benedikt Eberhardinger, Alexander Knapp, Wolfgang Reif, and Marcel Gehlen. 2016. Risk-based interoperability testing using reinforcement learning. In IFIP International Conference on Testing Software and Systems. Springer, 52--69.Google ScholarCross Ref
- Urban Richter, Moez Mnif, Jürgen Branke, Christian Müller-Schloer, and Hartmut Schmeck. 2006. Towards a generic observer/controller architecture for Organic Computing. GI Jahrestagung (1) 93 (2006), 112--119.Google Scholar
- Gregg Rothermel, Mary Jean Harrold, Jeffery Von Ronne, and Christie Hong. 2002. Empirical studies of test-suite reduction. Software Testing, Verification and Reliability 12, 4 (2002), 219--249.Google ScholarCross Ref
- Gregg Rothermel, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. 2001. Prioritizing test cases for regression testing. IEEE Transactions on software engineering 27, 10 (2001), 929--948. Google ScholarDigital Library
- August Shi, Alex Gyori, Milos Gligoric, Andrey Zaytsev, and Darko Marinov. 2014. Balancing trade-offs in test-suite reduction. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 246--256. Google ScholarDigital Library
- Joel A Tropp. 2009. Column subset selection, matrix factorization, and eigenvalue optimization. In Proceedings of the Twentieth Annual ACM-SIAM Symposium on Discrete Algorithms. Society for Industrial and Applied Mathematics, 978--986. Google ScholarDigital Library
- Lingming Zhang, Darko Marinov, Lu Zhang, and Sarfraz Khurshid. 2011. An empirical study of JUnit test-suite reduction. In Software Reliability Engineering (ISSRE), 2011 IEEE 22nd International Symposium on. IEEE, 170--179. Google ScholarDigital Library
Recommendations
Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction
Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization ...
Improving the effectiveness of test suite reduction for user-session-based testing of web applications
Context: Test suite reduction is the problem of creating and executing a set of test cases that are smaller in size but equivalent in effectiveness to an original test suite. However, reduced suites can still be large and executing all the tests in a ...
Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexity
Test suite reduction is an important approach that decreases the cost of regression testing. A test suite reduction technique operates based on the relationship between the test cases in the regression test suite and the test requirements in the program ...
Comments