Abstract
Testing should not reduce confidence in the system under test – unless defects are found. We show that for a general class of finite-state systems this intuition is incorrect. We base our argument on the view of risk as a probability. We calculate the risk of having an invalid implementation, based on a concrete, believable fault model, and show that executing correct test runs can actually decrease confidence in the system under test. This anomaly is important as it explains some of the difficulty in establishing mathematical links between fault models and testing efficiency. The presented anomaly itself is claimed to be independent of the particular structure of systems. We provide critique of the result, and discuss the potential limits of the presented anomaly as well as ways to remedy it.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Alur, R., Courcoubetis, C., Yannakakis, M.: Distinguishing tests for nondeterministic and probabilistic machines. In: Proc. 27th ACM Symposium on Theory of Computing, pp. 363–372 (1995)
European Telecommunications Standards Institute (ETSI). The testing and test control notation version 3; part 1: TTCN-3 core language. ETSI ES 201 873-1, V2.2.1 (February 2003)
Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines—a survey. Proceedings of the IEEE 84(8), 1090–1123 (1996)
Luo, G., von Bochmann, G., Petrenko, A.: Test selection based on communicating nondeterministic finite state machines using a generalized wp-method. IEEE Transactions on Software Engineering SE-20(2), 149–162 (1994)
Mas-Colell, A., Whinston, M.D., Green, J.R.: Microeconomic Theory. Oxford University Press, Oxford (1995)
Milton, J.S., Arnold, J.C.: Introduction to probability and statistics. McGraw–Hill, New York (1995)
Offutt, A.J.: Investigations of the software testing coupling effect. ACM Transactions on Software Engineering and Methodology 1(1), 5–20 (1992)
Offutt, A.J., Lee, S.: An empirical evaluation of weak mutation. IEEE Transactions on Software Engineering 20(5), 337–344 (1994)
Peterson, J.L.: Petri Net Theory and the Modeling of Systems. Prentice Hall PTR, Englewood Cliffs (1981)
Petrenko, A.: Fault model-driven test derivation from finite state models: Annotated bibliography. In: Cassez, F., Jard, C., Rozoy, B., Dermot, M. (eds.) MOVEP 2000. LNCS, vol. 2067, pp. 196–205. Springer, Heidelberg (2001)
Petrenko, A., Yevtushenko, N., Bochman, G.V.: Fault models for testing in context. In: FORTE 1996, pp. 163–178 (1996)
Petrenko, A., Yevtushenko, N., Huo, J.L.: Testing transition systems with input and output tester. In: Hogrefe, D., Wiles, A. (eds.) TestCom 2003. LNCS, vol. 2644, Springer, Heidelberg (2003)
Pyhälä, T., Heljanko, K.: Specification coverage aided test selection. In: Lilius, J., Balarin, F., Machado, R.J. (eds.) Proceeding of the 3rd International Conference on Application of Concurrency to System Design (ACSD 2003), pp. 187–195. IEEE Computer Society, Guimaraes, Portugal (2003)
Rimen, M., Ohlsson, J., Torin, J.: On microprocessor error behavior modeling. In: 24th IEEE International Symposium on Fault-Tolerant Computing (1994)
Tretmans, J.: A formal approach to conformance testing. In: Proc. 6th International Workshop on Protocols Test Systems, IFIP Transactions, vol. C-19, pp. 257–276 (1994)
van der Bijl, M., Rensink, A., Tretmans, J.: Compositional testing with IOCO. In: Petrenko, A., Ulrich, A. (eds.) FATES 2003. LNCS, vol. 2931, pp. 86–100. Springer, Heidelberg (2004)
Whittaker, J., Thomason, M.: A markov chain model for statistical software testing. IEEE Transactions on Software Engineering 20(10), 812–824 (1994)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Huima, A. (2006). A Note on an Anomaly in Black-Box Testing. In: Grieskamp, W., Weise, C. (eds) Formal Approaches to Software Testing. FATES 2005. Lecture Notes in Computer Science, vol 3997. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11759744_4
Download citation
DOI: https://doi.org/10.1007/11759744_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-34454-4
Online ISBN: 978-3-540-34455-1
eBook Packages: Computer ScienceComputer Science (R0)