Are test cases needed? Replicated comparison between exploratory and test-case-based software testing

Abstract

Manual software testing is a widely practiced verification and validation method that is unlikely to fade away despite the advances in test automation. In the domain of manual testing, many practitioners advocate exploratory testing (ET), i.e., creative, experience-based testing without predesigned test cases, and they claim that it is more efficient than testing with detailed test cases. This paper reports a replicated experiment comparing effectiveness, efficiency, and perceived differences between ET and test-case-based testing (TCT) using 51 students as subjects, who performed manual functional testing on the jEdit text editor. Our results confirm the findings of the original study: 1) there is no difference in the defect detection effectiveness between ET and TCT, 2) ET is more efficient by requiring less design effort, and 3) TCT produces more false-positive defect reports than ET. Based on the small differences in the experimental design, we also put forward a hypothesis that the effectiveness of the TCT approach would suffer more than ET from time pressure. We also found that both approaches had distinctive issues: in TCT, the problems were related to correct abstraction levels of test cases, and the problems in ET were related to test design and logging of the test execution and results. Finally, we recognize that TCT has other benefits over ET in managing and controlling testing in large organizations.

Appendices

Appendix A: Summary of the Survey Questions

1.1 Background

• Years of university studies

• Study credits

• How many years of experience do you have on the following areas?

  • Professional software development (any kind of role in development)

  • Professional programming (as a developer, programmer or equivalent)

  • Professional software testing (as a tester, developer, or equivalent)

  • Other kind of experience in software development

• Have you got any training on software testing before this course? (Yes or No)

  • What kind of training?

1.2 Coverage

• Assess the coverage of your testing on the following features

  • 4-step ordinal scale: not covered at all—covered superficially—basic functions well covered—covered thoroughly

1.3 Exploratory Approach

• How easy was the exploratory testing approach to apply in practice?

  • 7-step ordinal scale: (1) difficult … (4) neutral … (7) very easy

• How useful was the provided test charter for structuring and guiding your testing?

  • 7-step ordinal scale: (1) hinder … (4) neutral … (7) very useful

• How useful was the exploratory testing approach for finding defects?

  • 7-step ordinal scale: (1) hinder … (4) neutral … (7) very useful

• What problems or shortcomings did you experience in the exploratory testing approach?

1.4 Test-case Based Approach

• How easy were your own test cases to execute in practice?

  • 7-step ordinal scale: (1) difficult … (4) neutral … (7) very easy

• How useful were your own test cases for structuring and guiding your testing?

  • 7-step ordinal scale: (1) hinder … (4) neutral … (7) very useful

• How useful were your own test cases for finding defects?

  • 7-step ordinal scale: (1) hinder … (4) neutral … (7) very useful

• What problems or shortcomings did your test cases have?

• Which one of the two testing approaches (ET or TCT) gave you a better confidence to the quality of your testing, and why?

Appendix B: Contents of the ET Charter

• 1. What—tested areas

  Select the correct description of tested features for your exploratory testing and remove the other one:

  • Feature Set B1

    Search and replace (User’s Guide chapter 5)

    • Searching For Text

    • Replacing Text

      • Text Replace

    • HyperSearch

    • Multiple File Search

    • The Search Bar

    + Applicable shortcuts in Appendix A

  • Feature Set B2

    Editing source code (User’s Guide chapter 6)

    (test for one edit mode, e.g. java-mode)

    • Tabbing and Indentation

      • Soft Tabs

      • Automatic Indent

    • Commenting Out Code

    • Bracket Matching

    • Folding

      • Collapsing and Expanding Folds

      • Navigating Around With Folds

      • Miscellaneous Folding Commands

      • Narrowing

    + Applicable shortcuts in Appendix A

• 2. Why—goal and focus

  Perform testing from the viewpoint of a typical user and pay attention to following issues:

  • Does the function work as described in the user manual?

  • Does the function do any things that it should not do?

  • From the viewpoint of a typical user, does the function work as the user would expect and want?

  • What interactions the function has or might have with another functions, settings, data, or configuration of the application; do these interactions work correctly and as the user would expect and want them to work?

  Focus into functionality in your testing. Try to test exceptional cases, invalid as well as valid inputs, things that the user could do wrong, and typical error situations. However, do not test external and environment related (e.g. hardware) errors and exceptions (such as very low memory, broken hard drive, corrupted files, etc.).

• 3. How—approach

  Use the jEdit User’s Guide as the specification for the features, and utilize also your own knowledge and experience since the User’s Guide is neither comprehensive nor unambiguous. Use the following testing strategies for functional testing.

  Domain testing

  • equivalence partitioning

  • boundary value analysis

  Combination testing

  • Base choice strategy

  • Pair-wise (all-pairs) strategy

• 4. Exploration log

  SESSION START TIME: 2006-mm-dd hh:mm

  TESTER: _

  VERSION: jEdit 4.2 variant for T-76.5613 exercise

  ENVIRONMENT: _

  • 4.1 Task breakdown

    DURATION (hh:mm): __:__

    TEST DESIGN AND EXECUTION (percent): _%

    BUG INVESTIGATION AND REPORTING (percent): _%

    SESSION SETUP (percent): _%

  • 4.2 Test Data and Tools

    What data files and tools were used in testing?

  • 4.3 Test notes

    • Test notes that describe what was done, and how.

    • Detailed enough to be able to use in briefing the test session with other persons.

    • Detailed enough to be able to reproduce failures.

  • 4.4 Defects

    Time stamp, short note, Bugzilla bug ID

  • 4.5 Issues

    Any observations, issues, new feature requests and questions that came up during testing but were not reported as bugs.

Appendix C The Target Application and Features

The target of testing in the both experiments was jEdit open source text editor, version 4.2. with seeded defects in the tested features.

The official version and documentation of the target software can be accessed at: http://sourceforge.net/projects/jedit/files/jedit/4.2/

The user’s guide used as the source documentation for testing can be accessed at:

http://sourceforge.net/projects/jedit/files/jedit/4.2/jedit42manual-a4.pdf/download

The target feature sets used in the experiments were the following:

• Feature Set A (Used in the original experiment)

  • Working with files (User’s Guide chapter 4, pp. 11–12, 17)

    • Creating new files

    • Opening files (excluding GZipped files)

    • Saving files

    • Closing Files and Exiting jEdit

  • Editing text (User’s Guide chapter 5, 18–23)

    • Moving The Caret

    • Selecting Text

      • Range Selection

      • Rectangular Selection

      • Multiple Selection

    • Inserting and Deleting Text

    • Working With Words

      • What’s a Word?

    • Working With Lines

    • Working With Paragraphs

    • Wrapping Long Lines

      • Soft Wrap

      • Hard Wrap

    And the applicable shortcuts (User’s Guide Appendix A, pp. 46–50)

• Feature Set B1 (Used in the original and replicated experiment)

  • Search and replace (User’s Guide chapter 5, pp. 26–29)

    • Searching For Text

    • Replacing Text

      • Text Replace

    • HyperSearch

    • Multiple File Search

    • The Search Bar

    And the applicable shortcuts (User’s Guide Appendix A, pp. 46–50)

• Feature Set B2 (Used in the original and replicated experiment)

  • Editing source code (User’s Guide chapter 6, pp. 30–36)

    • Tabbing and Indentation

      • Soft Tabs

      • Automatic Indent

    • Commenting Out Code

    • Bracket Matching

    • Folding

      • Collapsing and Expanding Folds

      • Navigating Around With Folds

      • Miscellaneous Folding Commands

      • Narrowing

    And the applicable shortcuts (User’s Guide Appendix A, pp. 46–50)