Abstract
This paper presents dynamic testing, a method that exploits automata learning to systematically test (black box) systems almost without prerequisites. Based on interface descriptions and optional sample test cases, our method successively explores the system under test (SUT), in order to extrapolate a behavioural model. This is in turn used to steer the further exploration process. Due to the applied learning technique, our method is optimal in the sense that the extrapolated models are most concise (i.e. state minimal) in consistently representing all the information gathered during the exploration. Using the LearnLib, our framework for automata learning, our method can be elegantly combined with numerous optimisations of the learning procedure, with various choices of model structures, and with the option of dynamically/interactively enlarging the alphabet underlying the learning process. The latter is important in the Web context, where totally new situations may arise when following links. All these features are illustrated using as a case study the web application Mantis, a bug tracking system widely used in practice. In addition, we present another case study that demonstrates the scalability of the approach. We show how the dynamic testing procedure works and how behavioural models arise that concisely summarize the current testing effort. It turns out that these models reveal the system structure from a user perspective. Besides steering the automatic exploration process, they are ideal for user guidance and to support analyses to improve the system understanding, as they reveal the system structure from a user perspective.
Similar content being viewed by others
References
Raffelt, H., Steffen, B., Margaria, T.: Dynamic Testing Via Automata Learning. In: Proceedings of Haifa Verification Conference 2007, LNCS 4899, pp.136–152 (2007)
Raffelt, H., Margaria, T., Steffen, B., Merten, M.: Hybrid test of web applications with webtest. In: Proceedings of the 2008 Workshop on Testing, analysis, and verification of web services and applications (TAV-WEB ’08), pp. 1–7
Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. In Cerioli, M. (ed.) Proceedings of 8th International Conference on Fundamental Approaches to Software Engineering (FASE’05). vol. 3442 of LNCS, pp. 175–189. Springer, 4–8 April 2005
Raffelt, H., Steffen, B., Berg, T., Margaria, T.: LearnLib: a framework for extrapolating behavioral models. Int. J. Softw. Tools Tech. Transf. (2009). doi:10.1007/s10009-009-0111-8
Niese, O., Margaria, T., Hagerer, A., Steffen, B., Brune, G., Goerigk, W., Ide, H.D.: Automated regression testing of cti-systems. In: Proceedings of the IEEE European Test Workshop (ETW’01), p. 51. IEEE Computer Society, Washington, DC, USA (2001)
Margaria, T., Niese, O., Steffen, B., Erochok, A.: System level testing of virtual switch (re-)configuration over ip. In: Proceedings of the IEEE European Test Workshop (ETW’02), IEEE Computer Society Press, pp. 67–74 (2002)
Hagerer A., Margaria T., Niese O., Steffen B., Brune G., Ide H.D.: Efficient regression testing of cti-systems: testing a complex call-center solution. Ann. Rev. Commun. Int. Eng. Consortium (IEC), Chicago (USA) 55, 1033–1040 (2001)
Jörges, S., Kubczak, C., Nagel, R., Margaria, T., Steffen, B.: Model-driven development with the jabc. In: Proceedings of Haifa verification conference 2006 (HVC 2006). LNCS, Springer. http://www.springerlink.com/content/30v87502kn5h6672/
Margaria, T., Steffen, B., Reitenspieß, M.: Service-oriented design: The jabc approach. In: Proceedings of Service Oriented Computing (SOC 2005), Internationales Begegnungs- und Forschungszentrum für Informatik (IBFI), Schloss Dagstuhl, Germany (2005). http://dblp.uni-trier.de/rec/bibtex/conf/icsoc/MargariaSR05
Karusseit M., Margaria T.: Feature-based modelling of a complex, online-reconfigurable decision support service. Electr. Notes Theor. Comput. Sci. 157(2), 101–118 (2006)
Hungar, H., Margaria, T., Steffen, B.: Test-based model generation for legacy systems. In: Proceedings of 2003 International Test Conference (ITC 2003), pp. 971–980. IEEE Computer Society, Charlotte, NC (2003)
Margaria, T., Steffen, B.: Agile IT: thinking in user-centric models. In: Proceedings of ISoLA 2008, Leveraging Applications of Formal Methods, Verification and Validation, Third International Symposium, Porto Sani, Greece, October 13-15, 2008. Serie “Communications in Computer and Information Science” no. 17, pp. 490–502
Rational unified process. http://www-306.ibm.com/software/awdtools/rup/ (seen June 2007)
Hörmann, M., Margaria, T., Mender, T., Nagel, R., Schuster, M., Steffen, B., Trinh, H.: The jABC approach to rigorous collaborative development of SCM applications. In: Proceedings of ISoLA 2008, Leveraging Applications of Formal Methods, Verification and Validation, Third International Symposium, Porto Sani, Greece. Serie “Communications in Computer and Information Science” no. 17, pp. 724–737, 13–15 October 2008
Steffen, B., Narayan, P.: Full Life-Cycle Support for Endto- End Processes Computer. IEEE Computer Society Press, Los Alamitos (2007). http://portal.acm.org/citation.cfm?id=1320644
Raffelt, H., Steffen, B., Berg, T.: Learnlib: A library for automata learning and experimentation. In: Proc. of the 10th International Workshop on Formal Methods for Industrial Critical Systems (FMICS’05), pp. 62–71. ACM Press, Lisbon, Portugal (2005)
Mariani, L., Pezze, M.: Behaviour capture and test for controlling the quality of component-based integrated systems. In: Proceedings of ESEC/FSE Workshop on Tool Integration in System Development, Helsinki (2003)
Mercury TestDirector. http://www.fronde.com/products_mercury.htm (seen April 2008)
Ernst, M.D., Czeisler, A., Griswold, W.G., Notkin, D.: Quickly detecting relevant program invariants. In: Proceedings of 22th International Conference on Software Engineering (ICSE’00), pp. 449–458 (2000)
Nimmer, J.W., Ernst, M.D.: Automatic generation of program specifications. In: Proceedings of the 2002 International Symposium on Software Testing and Analysis (ISSTA’02), Rome, Italy, pp. 229–239, 22–24 July 2002
Brun, Y., Ernst, M.D.: Finding latent code errors via machine learning over program executions. In: Proceedings of the 26th International Conference on Software Engineering (ICSE’04), Edinburgh, Scotland, pp. 480–490 (2004)
Hagerer, A., Hungar, H., Niese, O., Steffen, B.: Model generation by moderated regular extrapolation. In: Kutsche, H.W.R., (ed.) Proceedings of the 5th International Conference on Fundamental Approaches to Software Engineering (FASE’02), vol. 2306 of LNCS., Heidelberg, Germany, Springer, pp. 80–95 (2002)
de la Higuera C.: A bibliographical study of grammatical inference. Pattern Recognit 38, 1332–1348 (2005)
Cook J.E., Wolf A.L.:: Discovering models of software processes from event-based data. (TOSEM) ACM Trans Softw Eng Methodol 7(3), 215–249 (1998)
Xie, T., Notkin, D.: Mutually enhancing test generation and specification inference. In: Petrenk, A., Ulrich, A. (eds.) Proceedings of 3rd International Workshop on Formal Approaches to Testing of Software (FATES’03). vol. 2931 of LNCS, pp. 60–69. Springer (2004)
Peled D., Vardi M.Y., Yannakakis M.: Black box checking. In: Wu, J., Chanson, S.T., Gao, Q. (eds) Proceedings of the Joint International Conference on Formal Description Techniques for Distributed System and Communication/Protocols and Protocol Specification, Testing and Verification FORTE/PSTV’99, pp. 225–240. Kluwer, Dordrecht (1999)
Cook, J.E., Du, Z., Liu, C., Wolf, A.L.: Discovering models of behavior for concurrent systems. Technical report, New Mexico State University, Department of Computer Science (2002) NMSU-CS-2002-010
Mariani, L., Pezzè, M.: A technique for verifying component-based software. In: Proceedings of International Workshop on Test and Analysis of Component Based Systems (TACoS’04), pp. 17–30 (2004)
Steffen, B., Margaria, T., Raffelt, H., Niese, O.: Efficient test-based model generation of legacy systems. In: Proceedings of the 9th IEEE International Workshop on High Level Design Validation and Test (HLDVT’04), Sonoma (CA), USA, IEEE Computer Society Press, pp. 95–100 (2004)
Margaria, T., Hinchey, M.G., Raffelt, H., Rash, J., Rouff, C.A., Steffen, B.: Completing and adapting models of biological processes. In: Proceedings of IFIP Conference on Biologically Inspired Cooperative Computing (BiCC 2006), Santiago (Chile). http://www.springerlink.com/content/2t8279g8g655u826/
Steffen, B., Hungar, H.: Behavior-based model construction. In: Mukhopadhyay, S., Zuck, L. (eds.) Proceedings of the 4th International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI’03). vol. 2575 of LNCS., Springer, pp. 5–19 (2003)
Angluin D.: Learning regular sets from queries and counterexamples. Inf. Comput. 2(75), 87–106 (1987)
Margaria T., Raffelt H., Steffen B.: Knowledge-based relevance filtering for efficient system-level test-based model generation. Innov. Syst. Softw. Eng. 1(2), 147–156 (2005)
Fujiwara S., von Bochmann G., Khendek F., Amalou M., Ghedamsi A.: Test selection based on finite state models. IEEE Trans. Softw. Eng. 17(6), 591–603 (1991)
Mantis bug tracker. http://www.mantisbt.org (seen June 2007)
Watij User Guide. http://watij.com/wiki:user_guide
Bhogal, K.S.: HttpUnit: A Civilized Way to Test Web Applications in WebSphere Studio. http://www.ibm.com/developerworks/websphere/library/techarticles/0303_bhogal/bhogal.html (2003)
Willcock, C., Dei T., Tobies, S., Keil, S., Engler, F., Schulz, S.: An Introduction to TTCN-3, First edition. Wiley (2005)
Troschütz, S.: Web Service Test Framework with TTCN-3 MSc. Thesis, University of Göttingen, June 2007. http://www.swe.informatik.uni-goettingen.de/publications/ST/WebServiceTestFrameworkWithTTCN-3.pdf
Testing technologies: TTworkbench Basic. http://testingtech.de/products/ttwb_basic.php
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was partially supported by the European Union Specific Targeted Research Project SHADOWS (IST-2006-35157), Self-Healing Approach to Designing cOmplex softWare Systems and the FET Project CONNECT: Emergent Connectors for Eternal Software Intensive Networked Systems (http://connect-forever.eu/).
This paper is an extended version of [1] and [2].
Rights and permissions
About this article
Cite this article
Raffelt, H., Merten, M., Steffen, B. et al. Dynamic testing via automata learning. Int J Softw Tools Technol Transfer 11, 307–324 (2009). https://doi.org/10.1007/s10009-009-0120-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10009-009-0120-7