Skip to main content
SpringerLink
Log in
Book cover

International Conference on Product-Focused Software Process Improvement

PROFES 2019: Product-Focused Software Process Improvement pp 299–314Cite as

Excellence in Exploratory Testing: Success Factors in Large-Scale Industry Projects

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11915))

Abstract

Based on interviews with 20 interviewees from four case study companies, this paper presents a list of key factors that enable efficient and effective exploratory testing of large-scale software systems. The nine factors are grouped into four themes: “The testers’ knowledge, experience and personality”, “Purpose and scope”, “Ways of working” and “Recording and reporting”. According to the interviewees, exploratory testing is a more creative way to work for the testers, and was therefore considered to make better use of the testers. Exploratory testing was also described as a good way to test system-wide and to test large-scale systems, especially exploratory testing with an end-user perspective. The identified key factors were confirmed by a series of follow-up interviews with the 20 interviewees and a cross-company workshop with 14 participants. This strengthens the generalizability of the findings, supporting that the list of key factors can be applied to projects in a large segment of the software industry. This paper also presents the results from a systematic literature review including 129 publications related to exploratory testing. No publication were found that summarizes the key factors that enable efficient and effective exploratory testing, which supports the novelty of the findings.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Gregory, J., Crispin, L.: More agile Testing. Addison Wesley, Boston (2015)

    Google Scholar 

  2. Hendrickson, E.: Explore It!, The Pragmatic Bookshelf (2013)

    Google Scholar 

  3. Mårtensson, T., Ståhl, D., Bosch, J.: Exploratory testing of large-scale systems – testing in the continuous integration and delivery pipeline. In: Felderer, M., Méndez Fernández, D., Turhan, B., Kalinowski, M., Sarro, F., Winkler, D. (eds.) PROFES 2017. LNCS, vol. 10611, pp. 368–384. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69926-4_26

    Chapter  Google Scholar 

  4. Kitchenham, B.: Procedures for performing systematic reviews. Keele UK Keele Univ. 33(2004), 1–26 (2004)

    Google Scholar 

  5. Robson, C., McCartan, K.: Real World Research, 4th edn. Wiley, Hoboken (2016)

    Google Scholar 

  6. Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Eng. 14(2), 131–164 (2009)

    Article  Google Scholar 

  7. Shah, S.M.A., Gencel, C., Alvi, U.S., Petersen, K.: Towards a hybrid testing process unifying exploratory testing and scripted testing. J. Softw. Evol. Process 26(2), 220–250 (2014)

    Article  Google Scholar 

  8. Thangiah, M., Basri, S.: A preliminary analysis of various testing techniques in Agile development - a systematic literature review. In: 3rd International Conference on Computer and Information Sciences, ICCOINS 2016, pp. 600–605 (2016)

    Google Scholar 

  9. Basri, S., Dominic, D.D., Murugan, T., Almomani, M.A.: A proposed framework using exploratory testing to improve software quality in SME’s. Advances in Intell. Syst. Comput. 843, 1113–1122 (2019)

    Google Scholar 

  10. Calpur, M.C., Arca, S., Calpur, T.C., Yilmaz, C.: Model dressing for automated exploratory testing. In: IEEE International Conference on Software Quality, Reliability and Security Companion, QRS-C 2017, pp. 577–578 (2017)

    Google Scholar 

  11. Frajtak, K., Bures, M., Jelinek, I.: Model-based testing and exploratory testing: is synergy possible? In: 6th International Conference on IT Convergence and Security, ICITCS 2016 (2016)

    Google Scholar 

  12. Gebizli, C.S., Sozer, H.: Improving models for model-based testing based on exploratory testing. In: 38th Annual International Computers, Software and Applications Conference Workshops, COMPSACW 2014, pp. 656–661 (2014)

    Google Scholar 

  13. Hellmann, T.D., Maurer, F.: Rule-based exploratory testing of graphical user interfaces. In: 2011 Agile Conference, Agile 2011, pp. 107–116 (2011)

    Google Scholar 

  14. Hudson, J., Denzinger, J.: Risk management for self-adapting self-organizing emergent multi-agent systems performing dynamic task fulfillment. Auton. Agents Multi-Agent Syst. 29(5), 973–1022 (2015)

    Article  Google Scholar 

  15. Kim, D.-K., Lee, L.-S.: Reverse engineering from exploratory testing to specification-based testing. Int. J. Softw. Eng. Appl. 8(11), 197–208 (2014)

    Google Scholar 

  16. Kuhn, A.: On extracting unit tests from interactive live programming sessions. In: International Conference on Software Engineering, pp. 1241–1244 (2013)

    Google Scholar 

  17. Mihindukulasooriya, N., Rizzo, G., Troncy, R., Corcho, O., García-Castro, R.: A two-fold quality assurance approach for dynamic knowledge bases: the 3cixty use case. In: CEUR Workshop Proceedings (2016)

    Google Scholar 

  18. Rashmi, N., Suma, V.: Defect detection efficiency of the combined approach. In: Satapathy, S., Avadhani, P., Udgata, S., Lakshminarayana, S. (eds.) ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India - Volume II. AISC, vol. 249, pp. 485–490. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-03095-1_51

    Chapter  Google Scholar 

  19. Schaefer, C.J., Do, H.: Model-based exploratory testing: a controlled experiment. In: 7th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2014, pp. 284–293 (2014)

    Google Scholar 

  20. Ghazi, A.N., Garigapati, R.P., Petersen, K.: Checklists to support test charter design in exploratory testing. In: Baumeister, H., Lichter, H., Riebisch, M. (eds.) XP 2017. LNBIP, vol. 283, pp. 251–258. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-57633-6_17

    Chapter  Google Scholar 

  21. Sviridova, T., Stakhova, D., Marikutsa, U.: Exploratory testing: management solution. In: 12th International Conference: The Experience of Designing and Application of CAD Systems in Microelectronics, CADSM 2013, p. 361 (2013)

    Google Scholar 

  22. Ghazi, A.N., Petersen, K., Bjarnason, E., Runeson, P.: Levels of exploration in exploratory testing: from freestyle to fully scripted. IEEE Access 6, 26416–26423 (2018)

    Article  Google Scholar 

  23. Raappana, P., Saukkoriipi, S., Tervonen, I., Mäntylä, M.V.: The effect of team exploratory testing - experience report from F-Secure. In: International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016, pp. 295–304 (2016)

    Google Scholar 

  24. Shah, S.M.A., Torchiano, M., Vetrò, A., Morisio, M.: Exploratory testing as a source of technical debt. IT Prof. 16(3), 44–51 (2014)

    Article  Google Scholar 

  25. Bures, M., Frajtak, K., Ahmed, B.S.: Tapir: automation support of exploratory testing using model reconstruction of the system under test. IEEE Trans. Reliab. 67(2), 557–580 (2018)

    Article  Google Scholar 

  26. Frajtak, K., Bures, M., Jelinek, I.: Exploratory testing supported by automated reengineering of model of the system under test. Cluster Comput. 20(1), 855–865 (2017)

    Article  Google Scholar 

  27. Reis, J., Mota, A.: Aiding exploratory testing with pruned GUI models. Inf. Process. Lett. 133, 49–55 (2018)

    Article  MathSciNet  Google Scholar 

  28. Gebizli, C.Ş., Sözer, H.: Automated refinement of models for model-based testing using exploratory testing. Softw. Qual. J. 25(3), 979–1005 (2017)

    Article  Google Scholar 

  29. Gebizli, C.S., Sözer, H.: Impact of education and experience level on the effectiveness of exploratory testing: an industrial case study. In: 10th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2017, pp. 23–28 (2017)

    Google Scholar 

  30. Itkonen, J., Mäntylä, M.V., Lassenius, C.: How do testers do it? An exploratory study on manual testing practices. In: 3rd International Symposium on Empirical Software Engineering and Measurement, ESEM 2009, pp. 494–497 (2009)

    Google Scholar 

  31. Itkonen, J., Mäntylä, M.V., Lassenius, C.: The role of the tester’s knowledge in exploratory software testing. IEEE Trans. Softw. Eng. 39(5), 707–724 (2013)

    Article  Google Scholar 

  32. Micallef, M., Porter, C., Borg, A.: Do exploratory testers need formal training? An investigation using HCI techniques. In: International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016, pp. 305–314 (2016)

    Google Scholar 

  33. Pfahl, D., Yin, H., Mäntylä, M.V., Münch, J.: How is exploratory testing used? A state-of-the-practice survey. In: International Symposium on Empirical Software Engineering and Measurement (2014)

    Google Scholar 

  34. Shoaib, L., Nadeem, A., Akbar, A.: An empirical evaluation of the influence of human personality on exploratory software testing. In: 13th International Multitopic Conference, INMIC 2009 (2009)

    Google Scholar 

  35. Tuomikoski, J., Tervonen, I.: Absorbing software testing into the scrum method. In: Bomarius, F., Oivo, M., Jaring, P., Abrahamsson, P. (eds.) PROFES 2009. LNBIP, vol. 32, pp. 199–215. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02152-7_16

    Chapter  Google Scholar 

  36. Pichler, J., Ramler, R.: How to test the intangible properties of graphical user interfaces? In: 1st International Conference on Software Testing, Verification and Validation, ICST 2008, pp. 494–497 (2008)

    Google Scholar 

  37. Kumar, S., Wallace, C.: Guidance for exploratory testing through problem frames. In: Software Engineering Education Conference, pp. 284–288 (2013)

    Google Scholar 

  38. Cook, T.D., Campbell, D.T., Day, A.: Quasi-Experimentation: Design & Analysis Issues for Field Settings, vol. 351. Houghton Mifflin, Boston (1979)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Saab AB, Linköping, Sweden

    Torvald Mårtensson

  2. University of Oslo, Oslo, Norway

    Antonio Martini

  3. Ericsson AB, Linköping, Sweden

    Daniel Ståhl

  4. Chalmers University of Technology, Gothenburg, Sweden

    Jan Bosch

Authors
  1. Torvald Mårtensson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonio Martini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel Ståhl
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jan Bosch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Torvald Mårtensson .

Editor information

Editors and Affiliations

  1. Universitat Politècnica de Catalunya, Barcelona, Spain

    Xavier Franch

  2. University of Helsinki, Helsinki, Finland

    Tomi Männistö

  3. Fraunhofer Institute for Experimental Software Engineering, Kaiserslautern, Germany

    Silverio Martínez-Fernández

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mårtensson, T., Martini, A., Ståhl, D., Bosch, J. (2019). Excellence in Exploratory Testing: Success Factors in Large-Scale Industry Projects. In: Franch, X., Männistö, T., Martínez-Fernández, S. (eds) Product-Focused Software Process Improvement. PROFES 2019. Lecture Notes in Computer Science(), vol 11915. Springer, Cham. https://doi.org/10.1007/978-3-030-35333-9_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-35333-9_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-35332-2

  • Online ISBN: 978-3-030-35333-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation