Abstract
The paper attempts to provide a comprehensive view of the field of software testing. The objective is to put all the relevant issues into a unified context, although admittedly the overview is biased towards my own research and expertise. In view of the vastness of the field, for each topic problems and approaches are only briefly tackled, with appropriate references provided to dive into them. I do not mean to give here a complete survey of software testing. Rather I intend to show how an unwieldy mix of theoretical and technical problems challenge software testers, and that a large gap exists between the state of the art and of the practice.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Adams, E.: Optimizing Preventive Service of Software Products. IBM J. of Research and Development 281 (1984) 2–14 4
Agedis: Automated Generation and Execution of Test Suites for DIstributed Component-based Software. <http://www.agedis.de/index.shtml> 8
Bache, R., Müllerburg, M.: Measures of Testability as a Basis for Quality Assurance. Software Engineering Journal. 5 (March 1990) 86–92 14
Baresi, L., Young, M.: Test Oracles. Tech. Report CIS-TR-01-02. On line at <http://www.cs.uoregon.edu/ michal/pubs/oracles.html> 11
Basanieri, F., Bertolino, A., Marchetti, E.: The Cow Suite Approach to Planning and Deriving Test Suites in UML Projects. Proc. 5th Int. Conf. UML 2002, Dresden, Germany. LNCS 2460 (2002) 383–397 8
Basili, V.R., Selby, R.W.: Comparing the Effectiveness of Software Testing Strategies. IEEE Trans. Software Eng. 13(12) (1987) 1278–1296 5, 6
Bernot, G., Gaudel M.C., Marre, B.: Software Testing Based On Formal Specifications: a Theory and a Tool. Software Eng. Journal 6 (1991) 387–405 8
Bertolino, A.: ISSTA 2002 Panel: Is ISSTA Research Relevant to Industrial Users?. In [36] 201–202 (see also following entries 203-209) 6
Bertolino, A.: Knowledge Area Description of Software Testing. Guide to the SWEBOK, Joint IEEE-ACM Software Engineering Coordinating Committee. (2001) On-line at: <http://www.swebok.org> 2, 6
Bertolino, A., Corradini, F, Inverardi, P., Muccini, H.: Deriving Test Plans from Architectural Descriptions. Proc. Int. ACM Conf. on Soft. Eng. (2000) 220–229 14
Bertolino, A., Inverardi, P., Muccini, H.: An Explorative Journey from Architectural Tests Definition downto Code Tests Execution. Proc. Int. Conf. on Soft. Eng. (2001) 211–220 9, 14
Bertolino, A., Marré, M.: A General Path Generation Algorithm for Coverage Testing. Proc. 10th Int. Soft. Quality Week, San Francisco, Ca. (1997) pap. 2T1 7
Bertolino, A., Polini, A.: Re-thinking the Development Process of Component-Based Software, ECBS 2002 Workshop on CBSE, Lund, Sweden (2002) 15
Bertolino, A., Polini, A.: WCT: a Wrapper for Component Testing. Proc. Fidji’2002, Luxembourg (to appear) (2002) 16
Bertolino, A., Strigini, L.: On the Use of Testability Measures for Dependability Assessment IEEE Trans. Software Eng. 22(2) (1996) 97–108 11, 14
Binder, R. V.: Testing Object-Oriented Systems-Models, Patterns, and Tools. Addison-Wesley (2000) 14, 15
Bochmann, G. V., Petrenko, A.: Protocol Testing: Review of Methods and Relevance for Software Testing. Proc. Int. Symp. on Soft. Testing and Analysis (ISSTA), Seattle (1994) 109–124 8
Börger, E.: The Origins and the Development of the ASM Method for High Level System Design and Analysis. J. of Universal Computer Science 8(1) (2002) 2–74. On line at: <http://www.jucs.org/jucs_8_1/the_origins_and_the> 1, 2
Briand, L., Labiche, Y.: A UML-Based Approach to System Testing. Software and Systems Modeling 1(1) (2002) 10–42 8
Brinksma, E., Tretmans, J.: Testing Transition Systems: An Annotated Bibliography. Proc. of MOVEP’2k, Nantes (2000) 44–50 8
Carver, R.H., Tai, K.-C.: Use of Sequencing Constraints for Specification-Based Testing of Concurrent Programs. IEEE Trans. on Soft. Eng. 24(6) (1998) 471–490 10
Choi, J-D., Zeller, A.: Isolating Failure-Inducing Thread Schedules. In [36] 210–220 4
Coward, P.D.: Symbolic Execution Systems-A Review. Software Eng. J. (1988) 229–239 7
Clarke, L.A., Richardson, D. J.: Applications of Symbolic Evaluation. The J. of Systems and Software 5 (1985) 15–35 2
Crnkovic, I.: Component-based Software Engineering-New Challenges in Software Development. John Wiley&Sons (2001) 15
DeMillo, R.A., Offutt, A. J.: Constraint-Based Test Data Generation. IEEE Trans. Software Eng. 17(9) (1991) 900–910 8
Dick, J., Faivre, A.: Automating The Generation and Sequencing of Test Cases From Model-Based Specifications. Proc. FME’93, LNCS 670 (1993) 268–284 8, 9
Dijkstra, E.W.: Notes on Structured Programming. T.H. Rep. 70-WSK03 (1970) On line at: <http://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF> 1, 3
Duesterwald, E., Gupta, R., Soffa, M. L.: Rigorous Data Flow Testing Through Output Influences. Proc. 2nd Irvine Software Symposium, Irvine, CA. (1992) 131–145 7
Duran, J. W., Ntafos, S.C.: An Evaluation of Random Testing. IEEE Trans. Software Eng. SE-10(4) (1984) 438–444 5
Dunsmore, A., Roper, M., Wood, M.: Further Investigations into the Development and Evaluation of Reading Techniques for Object-Oriented Code Inspection. Proc. 24th Int. Conf. on Soft. Eng. Orlando, FL, USA (2002) 47–57 2
Edelstein, O., Farchi, E., Nir, Y., Ratsaby, G., Ur, S.: Multithreaded Java Program Test Generation. IBM Systems Journal 41 (2002) 111–125 10
Fagan, M.R.: Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal 15(3) (1976) 182–211 2
Fenton, N.E., Ohlsson, N.: Quantitative Analysis of Faults and Failures in a Complex Software System. IEEE Trans. Software Eng. 26(8) (2000) 797–814 4
Forgacs, I., Bertolino, A.: Preventing Untestedness in Data-flow Based Testing. Soft. Testing, Verification and Reliability 12(1)(2001) 29–61 7
Frankl, P.G. (Ed.): Proc. ACM Sigsoft Int. Symposium on Soft. Testing and Analysis ISSTA 2002, Soft. Engineering Notes 27(4) Roma, Italy (July 2002) 17, 18, 19
Frankl, P.G., Hamlet, R.G., Littlewood, B., Strigini, L.: Evaluating Testing Methods by Delivered Reliability. IEEE Trans. Software Eng. 24(8) (1998) 586–601 4, 5
Freedman, R. S.: Testability of Software Components. IEEE Trans. Software Engineering 17(6) (1991) 553–564 14
Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns-Elements of Reusable Object-Oriented Software. Addison-Wesley (1994) 15
Gao, J.: Component Testability and Component Testing Challenges, Proc. ICSE Workshop on Component-Based Soft. Eng. (2000) http://www.sei.cmu.edu/cbs/cbse2000/paper/ 15
Gao, J., Gupta, K., Gupta, S., Shim, S.: On Building Testable Software Components. Proc. ICCBSS2002, LNCS 2255 (2002) 108–121 16
Gargantini, A., Riccobene, E.: ASM-Based Testing: Coverage Criteria and Automatic Test Sequence. J. of Universal Computer Science 7(11) (2001) 1050–1067. On line at: <http://www.jucs.org/jucs_7_11/asm_based_testing_coverage> 2
Grieskamp, W., Gurevich, Y., Schulte, W., Veanes, M.: Generating Finite State Machines from Abstract State Machines. In [36] 112–122 2
Hamlet, D.: Continuity in Software Systems. In [36] 196–200 1
Hamlet, D., Taylor, R.: Partition Testing Does Not Inspire Confidence. IEEE Trans. Software Eng. 16(12) (1990) 1402–1411 5, 12
Harrold, M. J.: Testing: A Roadmap. In A. Finkelstein (Ed.), The Future of Software Engineering, ACM Press (2000) 63–72 17
Harrold, M. J., Soffa, M. L.: Selection of Data for Integration Testing.IEEE Software (March 1991) 58–65 14
Hierons, R. M.: Testing from a Z Specification. Soft. Testing, Verification and Reliability 7(1997) 19–33 8
Hierons, R., Derrick, J. (Eds): Special Issue on Specification-based Testing. Soft. Testing, Verification and Reliability 10 (2000) 8
Hiller, M., Jhumka, A., Suri, N.: PROPANE: An Environment for Examining the Propagation of Errors in Software. In [36] 81–85 4
Howden, W.E.: Weak Mutation Testing and Completeness of Test Sets. IEEE Trans. Software Eng. 8(4) (1982) 371–379 4
ISO/IEC 9126, Information Technology-Software Product Evaluation-Quality Characteristics and Guidelines for Their Use. (1991) 13
Fernandez, J.-C., Jard, C., Jeron, T., Nedelka, L., Viho, C.: Using On-the-fly Verification Techniques for the Generation of Test Suites. Proc. of the 8th Int. Conf. on Computer Aided Verification (1996) 8
Korel, B.: Automated Software Test Data Generation. IEEE Trans. Software Eng. 16(8) (1990) 870–879 7
Laprie, J.C.: Dependability-Its Attributes, Impairments and Means. In [66] 3–18 3
Latella, D., Massink, M.: On Testing and Conformance Relations for UML Statechart Diagrams Behaviours. In [36] 144–153 8
Littlewood, B., Popov, P. T., Strigini, L., Shryane, N.: Modeling the Effects of Combining Diverse Software Fault Detection Techniques. IEEE Trans. Software Eng. 26(12) (2000) 1157–1167 5
Littlewood, B., Strigini, L.: Validation of Ultra-High Dependability for Softwarebased Systems. Communications of the ACM. 36(11) (1993) 69–80 13
Lyu, M.R. (Ed.): Handbook of Software Reliability Engineering. IEEE Comp. Soc. Press/McGraw-Hill (1996) 8, 12
Morell, L. J.: A Theory of Fault-based Testing. IEEE Trans. Software Eng. 16(8) (1990) 844–857 4
Morris, J., Lee, G., Parker, K., Bundell, G. A., Lam, C. P.: Software Component Certification. IEEE Computer (September 2001) 30–36 16
Myers, G. J.: The Art of Software Testing. Wiley. (1979) 7, 8
Orso, A., Harrold, M. J., Rosenblum, D.: Component Metadata for Software Engineering Tasks. Proc. EDO2000, LNCS 1999 (2000) 129–144 16
Ostrand, T. J., Balcer, M. J.: The Category-Partition Method for Specifying and Generating Functional Tests. ACM Comm. 31(6) (1988) 676–686 8
Pargas, R., Harrold, M. J., Peck, R.: Test-Data Generation Using Genetic Algorithms. J. of Soft. Testing, Verifications, and Reliability 9 (1999) 263–282 7
Randell, B., Laprie, J. C., Kopetz, H., Littlewood B., Eds.: Predictably Dependable Computing Systems, Springer (1995) 19, 20
Rapps, S., Weyuker, E. J.: Selecting Software Test Data Using Data Flow Information. IEEE Trans. Software Eng., SE-11 (1985) 367–375 7
Richardson, D.J, Clarke, L.A.: Partition Analysis: A Method Combining Testing and Verification. IEEE Trans. Software Eng., SE-11 (1985) 1477–1490 5
Richardson, D., Thompson, M.C.: The Relay Model for Error Detection and its Application. Proc. 2nd Wksp Soft. Testing, Verification, and Analysis. Ban. Alberta, ACM/Sigsoft and IEEE July 1988 223–230 4
Rothermel, G., Harrold, M. J.: Analyzing Regression Test Selection Techniques. IEEE Trans. Software Eng., 22(8) (1996) 529–551 15
Stafford, J. A., Wolf, A. L.: Annotating Components to Support Component-Based Static Analyses of Software Systems, Proc. Grace Hopper Celeb. of Women in Computing (2001) 16
TESTNET-Integration of Testing Methodologies. <http://http://www-lor.int-evry.fr/testnet/> 13
TGV-Test Generation from transitions systems using Verification techniques. On line at <http://www.inrialpes.fr/vasy/cadp/man/tgv.html> 8
Thevenod-Fosse P., Waeselynck H., Crouzet Y.: Software Statistical Testing. In [66] 253–272 5
J. Tretmans. Conformance Testing with Labeled Transition Systems: Implementation Relations and Test Generation. Computer Networks and ISDN Systems 29 (1996) 49–79 8
Vegas, S.: Characterisation Schema for Selecting Software Testing Techniques. Int. Software Engineering Research Network (ISERN’01) Strachclyde, Scotland (2001) 3, 6
Voas, J.M.: PIE: A Dynamic Failure-Based Technique. IEEE Trans. Software Eng. 18(8) (1992) 717–727 4
Voas, J.M., Miller, K.W.: Software Testability: The New Verification. IEEE Software (May 1995) 17–28 14
Voas, J.: Certifying O.-the-Shelf Software Components. IEEE Computer (June 1998) 53–59 16
Voas, J.: Developing a Usage-Based Software Certification Process. IEEE Computer (August 2000) 32–37 16
Wang, Y., King, G., Wickburg, H.: A Method for Built-in Tests in Componentbased Software Maintenance, Proc. European Conference on Soft. Maintenance and Reengineering (1998) 186–189 16
Weyuker, E. J.: Translatability and Decidability Questions for Restricted Classes of Program Schemas. SIAM J. on Computers 8(4) (1979) 587–598 7
Weyuker, E. J.: On Testing Non-testable Programs. The Computer Journal 25(4) (1982) 465–470 10
Weyuker, E. J., Jeng, B.: Analyzing Partition Testing Strategies. IEEE Trans. Software Eng. 17(7) (1991) 703–711 5
Weyuker, E. J., Ostrand, T. J.: Theories of Program Testing and the Application of Revealing Subdomains. IEEE Trans. Software Eng. SE-6(1980) 236–246 5
Wood, M., Roper, M., Brooks, A., Miller, J.: Comparing and Combining Software Defect Detection Techniques: A Replicated Empirical Study. Proc. ESEC/FSE, LNCS 1301 (1997) 262–277 5
Zhu, H., Hall, P.A.V., May, J.H.R.: Software Unit Test Coverage and Adequacy. ACM Computing Surveys, 29 (1997) 366–427 4
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bertolino, A. (2003). Software Testing Research and Practice. In: Börger, E., Gargantini, A., Riccobene, E. (eds) Abstract State Machines 2003. ASM 2003. Lecture Notes in Computer Science, vol 2589. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36498-6_1
Download citation
DOI: https://doi.org/10.1007/3-540-36498-6_1
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-00624-4
Online ISBN: 978-3-540-36498-6
eBook Packages: Springer Book Archive