Abstract
In this paper, we show how our approach to active continuous quality control (ACQC), which employs learning technology to automatically maintain test models along the whole life cycle, can be extended to include risk analysts for supporting risk-based testing. Key to this enhancement is the tailoring of ACQC’s characteristic automata learning-based model extraction to prioritize critical aspects. Technically, risk analysts are provided with an abstract modeling level tailored to design test components (learning symbols) that encompass data-flow constraints reflecting a given risk profile. The resulting alphabet models are already sufficient to steer the ACQC process in a fashion that increases the risk coverage, while it at the same time radically reduces the testing effort. We illustrate our approach by means of case studies with Springer’s Online Conference Service (OCS) which show the impact of the risk prioritization on the performance: risk-based regression testing tailored for system migration and for pure functional evolution.



















Similar content being viewed by others
Notes
This case study of the router with its over 20.000 states was a first indication of the scalability of learning-based validation.
Java Platform, Enterprise Edition. For more information please see http://www.oracle.com/technetwork/java/javaee/.
The Java Platform, Enterprise Edition 6 was released in December 2009.
In the active automata learning context, the term system under learning (SUL) is used.
The executions on the SUT are denoted by test cases, the corresponding models are denoted by test case models, and the abstract input word is denoted by membership query.
LearnLib is available at http://www.learnlib.de.
From now on we do not consider ‘maximal reuse’ as we realized it on the (learn-) algorithm level by reusing the distinguishing suffixes. This way we avoid equivalence queries as we switch to a new application version of the system under learning. Fortunately, this feature is implemented once and may be used for every incremental learn setup.
CDI is defined in JSR 299, for a short introduction please see http://www.oracle.com/technetwork/articles/javaee/javaee6overview-141808.html#webbeans.
For the first input symbol the ‘reset’ activity is selected as predecessor since the ‘reset’ is already connected to the start activity (cf. Fig. 6).
Before the bidding and assignment phase started only after the upload phase had been ended.
A corresponding check had been unnecessary, because previously the upload phase had to be terminated before the assignment phase could be started.
References
Aarts, F., Jonsson, F., Uijen, J.: Generating models of infinite-state communication protocols using regular inference with abstraction. In: ICTSS ’10, Volume 6435 of LNCS, pp. 188–204. Springer, Berlin (2010)
Aarts, F., Schmaltz, J., Vaandrager, F.: Inference and abstraction of the biometric passport. In: ISoLA 2010, Volume 6415 of LNCS, pp. 673–686. Springer, Berlin (2010)
Angluin, D.: Learning regular sets from queries and counterexamples. Inf. Comput. 75(2), 87–106 (1987)
Bach, J.: Heuristic risk-based testing. Softw. Test. Quality Eng. Mag. 11, 99 (1999)
Bauer, O., Neubauer, J., Steffen, B., Howar, F.: Reusing system states by active learning algorithms. In: Moschitti, A., Scandariato, R. (eds.) Eternal Systems, Volume 255 of CCSE. Springer, Berlin (2012)
Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. In: FASE ’05, Volume 3442 of LNCS, pp. 175–189. Springer, Berlin (2005)
Beydeda, S., Gruhn, V.: Integrating white- and black-box techniques for class-level regression testing. In: COMPSAC ’01, Washington, DC, USA, pp. 357–362. IEEE Computer Society (2001)
Bossert, V., Hiet, G., Henin, G.: Modelling to simulate botnet command and control protocols for the evaluation of network intrusion detection systems. In: SAR-SSI ’11, pp. 1–8. IEEE Computer Society (2011)
Chen, Y., Probert, R.L., Ural, H.: Model-based regression test suite generation using dependence analysis. In: A-MOST ’07, New York, NY, USA, pp. 54–62. ACM (2007)
Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. 4(3), 178–187 (1978)
Felderer, M., Haisjackl, M., Breu, R., Motz, J.: Integrating manual and automatic risk assessment for risk-based testing. In: Biffl, S., Winkler, D., Bergsmann, J. (eds.) Software Quality, Process Automation in Software Development, Volume 94 of Lecture Notes in Business Information Processing, pp. 159–180. Springer, Berlin (2012)
Felderer, M., Ramler, R.: A multiple case study on risk-based testing in industry. sttt-RBT, Under, Review (2014)
Fowler, M.: Inversion of control containers and the dependency injection pattern (2004) . http://www.martinfowler.com/articles/injection.html
Gerrard, P., Thompson, N.: Risk based e-business resting. Artech House (2002)
Gnesi, S., Margaria, T.: Formal Methods for Industrial Critical Systems: A Survey of Applications. Wiley, :London (2012)
Grabowski, J., Hogrefe, D., Réthy, G., Schieferdecker, I., Wiles, A., Willcock, C.: An introduction to the testing and test control notation (ttcn-3). Comput. Netw. 42(3), 375–403 (2003)
Grabowski, J., Wiles, A., Willcock, C., Hogrefe, D.: On the design of the new testing language ttcn-3. In: Ural, H., Probert, R., Bochmann, G. (eds.) Testing of Communicating Systems, Volume 48 of IFIP Advances in Information and Communication Technology, pp. 161–176. Springer, New York (2000)
Grossmann, J.: Testing hybrid systems with ttcn-3 embedded. sttt-RBT. Under Review (2014)
Hagerer, A. Hungar, H., Niese, O., Steffen, B.: Model generation by moderated regular extrapolation. In FASE ’02, pp. 80–95, London, UK. Springer, Berlin (2002)
Howar, F., Steffen, B., Jonsson, B., Cassel, S.: Inferring canonical register automata. In: Kuncak, V., Rybalchenko, A. (eds.) Verification, Model Checking, and Abstract Interpretation, Volume 7148 of Lecture Notes in Computer Science, pp. 251–266. Springer, Berlin (2012)
Howar, F., Steffen, B., Merten, M.: Automata learning with automated alphabet abstraction refinement. In: Verification, Model Checking, and Abstract Interpretation, Volume 6538 of LNCS, pp. 263–277. Springer, Berlin (2011)
Hungar, H., Niese, O., Steffen, B.: Domain-specific optimization in automata learning. In: Computer Aided Verification, Volume 2725 of LNCS, pp. 315–327. Springer, Berlin (2003)
Jörges, S., Margaria, T., Steffen, B.: Genesys: service-oriented construction of property conform code generators. ISSE 4(4), 361–384 (2008)
Jéron, T.: Symbolic model-based test selection. Electron. Notes Theor. Comput. Sci. 240(0):167–184 (2009). Proceedings of the Eleventh Brazilian Symposium on Formal Methods (SBMF 2008)
Jörges, S.: Construction and Evolution of Code Generators: A Model-Driven and Service-Oriented Approach. Springer, Berlin (2013)
Karusseit, M., Margaria, T.: Feature-based modelling of a complex, online-reconfigurable decision support service. In WWV ’05, March 2005. ENTCS 1132
Korel, B., Al-Yami, A.M.: Automated regression test generation. SIGSOFT Softw. Eng. Notes 23(2), 143–152 (1998)
Lund, M.S., Solhaug, B.: Model-Driven Risk Analysis. Springer, Berlin (2011)
Marchand, H., Dubreil, J., Jéron, T.: Automatic testing of access control for security properties. In: Núñez, M., Baker, P., Merayo, M. (eds.) Testing of Software and Communication Systems, Volume 5826 of Lecture Notes in Computer Science, pp. 113–128. Springer, Berlin (2009)
Margaria, T., Niese, O., Raffelt, H., Steffen. B.: Efficient test-based model generation for legacy reactive systems. In: HLDVT ’04, Washington, DC, USA, pp. 95–100. IEEE Computer Society (2004)
Margaria, T., Steffen, B.: Lightweight coarse-grained coordination: a scalable system-level approach. STTT 5(2–3), 107–123 (2004)
Margaria, T., Steffen, B.: Agile IT: thinking in user-centric models. In: ISoLA 2009, Volume 17 of Communications in Computer and Information Science, pp. 490–502. Springer, Berlin (2009)
Margaria, T., Steffen, B.: Service-orientation: conquering complexity with XMDD. In: Hinchey, M., Coyle, L. (eds.) Conquering Complexity, pp. 217–236. Springer, London (2012)
Merten, M., Steffen, B., Howar, F., Margaria T.: Next generation learnlib. In: TACAS 2011, Volume 6605 of LNCS, pp. 220–223. Springer, Berlin (2011)
Müller-Olm, M., Schmidt, D., Steffen, B.: Model-checking: a tutorial introduction. In: Cortesi, A., Filé, G. (eds.) Static Analysis, Volume 1694 of LNCS, pp. 330–354. Springer, Berlin (1999)
Neubauer, J., Steffen, B.: Plug-and-play higher-order process integration. Computer 46(11), 56–62 (2013)
Neubauer, J., Steffen, B.: Learning-based cross-platform conformance testing . STVR. Wiley (2014)
Neubauer, J. Steffen, B., Margaria, T.: Higher-order process modeling: product-lining, variability modeling and beyond. arXiv, preprint arXiv:1309.5143 (2013)
Pickin, S., Jard, C., Jéron, T., Jézéquel, J.-M., Traon, Y.L.: Test synthesis from uml models of distributed software. IEEE Trans. Softw. Eng. 33(4), 252–269 (2007)
Raffelt, H., Merten, M., Steffen, B., Margaria, T.: Dynamic testing via automata learning. STTT 11(4), 307–324 (2009)
Raffelt, H., Steffen, B., Berg, T., Margaria, T.: LearnLib: a framework for extrapolating behavioral models. Int. J. Softw. Tools Technol. Transf. 11(5), 393–407 (2009)
Rivest, R.L., Schapire, R.E.: Inference of finite automata using homing sequences. Inf. Comput. 103(2), 299–347 (1993)
Rosenberg, L., Stapko, R., Gallo. A.: Risk-based object oriented testing. In: Proceedings of 13th International Software/Internet Quality Week-QW, San Francisco, California, USA, Vol. 2 (2000)
Rothermel, G., Harrold, M.J., Dedhia, J.: Regression test selection for C++ software. Softw. Test. Verif. Reliab. 10, 2000 (1999)
Shahbaz, M., Shashidhar, K.C., Eschbach. R.: Iterative refinement of specification for component based embedded systems. In: ISSTA ’11, pp. 276–286. ACM (2011)
Smolka, S., Steffen, B.: Priority as extremal probability. Formal Asp. Comput. 8(5), 585–606 (1996)
Steffen, B.: Characteristic formulae. In: ICALP, Volume 372 of LNCS, pp. 723–732 (1989)
Steffen, B., Howar, F., Merten. M.: Introduction to active automata learning from a practical perspective. In: Bernardo, M., Issarny, V. (eds.) Formal Methods for Eternal Networked Software Systems, Volume 6659 of LNCS, pp. 256–296. Springer, Berlin (2011)
Steffen, B., Margaria, T., Nagel, R., Jörges, S., Kubczak, C.: Model-driven development with the jABC. In: Bin, E., Ziv, A., Ur, S. (eds.) Hardware and Software. Verification and Testing, Volume 4383 of Lecture Notes in Computer Science, pp. 92–108. Springer, Berlin (2007)
Stepien, B., Peyton, L.: Innovation and evolution in integrated web application testing with ttcn-3. sttt-RBT. Under Review (2014)
Tretmans, J.: Model based testing with labelled transition systems. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing, Volume 4949 of LNCS, pp. 1–38. Springer, Berlin (2008)
Tretmans, J.: Model-based testing and some steps towards test-based modelling. In: Bernardo, M., Issarny, V. (eds.) Formal Methods for Eternal Networked Software Systems, Volume 6659 of LNCS, pp. 297–326. Springer, Berlin (2011)
Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22(5), 297–312 (2012)
Windmüller, S., Neubauer, J., Steffen, B., Howar, F., Bauer, O.: Active continuous quality control. In: Proceedings of the 16th International ACM Sigsoft symposium on Component-Based Software Engineering, CBSE ’13, New York, NY, USA, pp. 111–120. ACM (2013)
Xu, L., Dias, M., Richardson, D.: Generating regression tests via model checking. In: COMPSAC ’04, Washington, DC, USA, pp. 336–341. IEEE Computer Society (2004)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Neubauer, J., Windmüller, S. & Steffen, B. Risk-based testing via active continuous quality control. Int J Softw Tools Technol Transfer 16, 569–591 (2014). https://doi.org/10.1007/s10009-014-0321-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10009-014-0321-6