Skip to main content
SpringerLink
Log in

Artificial Intelligence Techniques in System Testing

  • Chapter
  • First Online:
Optimising the Software Development Process with Artificial Intelligence

Part of the book series: Natural Computing Series ((NCS))

Abstract

System testing is essential for developing high-quality systems, but the degree of automation in system testing is still low. Therefore, there is high potential for Artificial Intelligence (AI) techniques like machine learning, natural language processing, or search-based optimization to improve the effectiveness and efficiency of system testing. This chapter presents where and how AI techniques can be applied to automate and optimize system testing activities. First, we identified different system testing activities (i.e., test planning and analysis, test design, test execution, and test evaluation) and indicated how AI techniques could be applied to automate and optimize these activities. Furthermore, we presented an industrial case study on test case analysis, where AI techniques are applied to encode and group natural language into clusters of similar test cases for cluster-based test optimization. Finally, we discuss the levels of autonomy of AI in system testing.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Editors note: The source code linked in this chapter could change after publishing this book. A snapshot of the source code accompanying this chapter can be found at https://doi.org/10.5281/zenodo.6965479.

  2. 2.

    Please note that the authors have approval from the third party for the utilized case study and its code provided in the Git repository https://github.com/leohatvani/clustering-dependency-detection. The permission to use the code is also granted to the reader.

  3. 3.

    The source code of our work can be found online at [16], together with anonymized feature vectors and a test case graph.

  4. 4.

    Is a measure of a model’s accuracy on a dataset.

  5. 5.

    The Area Under Curve provides an aggregate measure of performance across all possible classification thresholds.

  6. 6.

    Based on the levels shown by Synopsis https://www.synopsys.com/automotive/autonomous-driving-levels.html.

References

  1. A. Abran, J. Moore, P. Bourque, R. Dupuis, L. Tripp, Software engineering body of knowledge. IEEE Comput. Soc. (2004)

    Google Scholar 

  2. D. Adamo, M.K. Khan, S. Koppula, R. Bryce, Reinforcement learning for android GUI testing. In: Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation (2008), pp. 2–8

    Google Scholar 

  3. S. Anand, E. Burke, T.Y. Chen, J. Clark, M. Cohen, W. Grieskamp, M. Harman, M. Harrold, P. McMinn, A. Bertolino et al., An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86(8), 1978–2001 (2013)

    Article  Google Scholar 

  4. A. Arcuri, Test suite generation with the many independent objective (MIO) algorithm. Inf. Softw. Technol. 104, 195–206 (2018)

    Article  Google Scholar 

  5. K. Baral, J. Offutt, F. Mulla, Self determination: a comprehensive strategy for making automated tests more effective and efficient, in 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) (IEEE, 2021), pp. 127–136

    Google Scholar 

  6. G. Bath, E. Van Veenendaal, Improving the Test Process: Implementing Improvement and Change-A Study Guide for the ISTQB Expert Level Module (Rocky Nook, Inc., 2013)

    Google Scholar 

  7. L.C. Briand, Y. Labiche, M. Shousha, Using genetic algorithms for early schedulability analysis and stress testing in real-time systems. Genet. Program Evolvable Mach. 7(2), 145–170 (2006)

    Article  Google Scholar 

  8. P. Dutta, G. Ryan, A. Zieba, S. Stolfo, Simulated user bots: real time testing of insider threat detection systems, in 2018 IEEE Security and Privacy Workshops (SPW) (IEEE, 2018), pp. 228–236

    Google Scholar 

  9. E. Enoiu, M. Frasheri, Test agents: The next generation of test cases, in International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (IEEE, 2019), pp. 305–308

    Google Scholar 

  10. L. Erlenhov, F.G. Oliveira Neto, R. Scandariato, P. Leitner, Current and future bots in software development, in International Workshop on Bots in Software Engineering (BotSE) (IEEE, 2019), pp. 7–11

    Google Scholar 

  11. M. Felderer, I. Schieferdecker, A taxonomy of risk-based testing. Int. J. Softw. Tools Technol. Transfer 16(5), 559–568 (2014)

    Article  Google Scholar 

  12. R. Feldt, F.G. Oliveira Neto, R. Torkar, Ways of applying artificial intelligence in software engineering, in International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE) (IEEE, 2018), pp. 35–41

    Google Scholar 

  13. K. Frounchi, L.C. Briand, L. Grady, Y. Labiche, R. Subramanyan, Automating image segmentation verification and validation by learning test oracles. Inf. Softw. Technol. 53(12), 1337–1348 (2011)

    Article  Google Scholar 

  14. V. Garousi, M. Felderer, Ç.M. Karapıçak, U. Yılmaz, Testing embedded software: a survey of the literature. Inf. Softw. Technol. 104, 14–45 (2018)

    Article  Google Scholar 

  15. M. Harman, P. McMinn, J.T. Souza, S. Yoo, Search based software engineering: techniques, taxonomy, tutorial, in Empirical Software Engineering and Verification (Springer, Berlin, 2010), pp. 1–59

    Google Scholar 

  16. L. Hatvani, S. Tahvili, Clustering dependency detection (2018). https://github.com/leohatvani/clustering-dependency-detection

  17. IEEE: IEEE standard glossary of software engineering terminology. IEEE Std. 610.12-1990 (1990), pp. 1–84

    Google Scholar 

  18. J.Y. Jiang, M. Zhang, C. Li, M. Bendersky, N. Golbandi, M. Najork, Semantic text matching for long-form documents, in WWW ’19 (Association for Computing Machinery, New York, NY, USA, 2019)

    Google Scholar 

  19. M. Johnson, J. Bradshaw, P. Feltovich, C. Jonker, B. Van Riemsdijk, M. Sierhuis, The fundamental principle of coactive design: Interdependence must shape autonomy, in International Workshop on Coordination, Organizations, Institutions, and Norms in Agent Systems (Springer, Berlin, 2010), pp. 172–191

    Google Scholar 

  20. A. Joshi, E. Fidalgo, E. Alegre, L. Fernández-Robles, Summcoder: an unsupervised framework for extractive text summarization based on deep auto-encoders. Expert Syst. Appl. 129, 200–215 (2019)

    Article  Google Scholar 

  21. M. Kane, Validating the interpretations and uses of test scores. J. Educ. Meas. 50, 1–73 (2013)

    Article  Google Scholar 

  22. P. Kumaresen, M. Frasheri, E. Enoiu, Agent-based software testing: a definition and systematic mapping study, in 2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C) (IEEE, 2020), pp. 24–31

    Google Scholar 

  23. D. Liang, F. Zhang, W. Zhang, Q. Zhang, J. Fu, M. Peng, T. Gui, X. Huang, Adaptive multi-attention network incorporating answer information for duplicate question detection, in SIGIR’19 (Association for Computing Machinery, New York, NY, USA, 2019), pp. 95–104

    Google Scholar 

  24. L. van der Maaten, E. Postma, H. Herik, Dimensionality reduction: a comparative review. J. Mach. Learn. Res.-JMLR 10 (2007)

    Google Scholar 

  25. C. Malz, N. Jazdi, Agent-based test management for software system test, in International Conference on Automation Quality and Testing Robotics (AQTR), vol. 2 (IEEE, 2010), pp. 1–6

    Google Scholar 

  26. M. Mansoor, Z. Rehman, M. Shaheen, M. Khan, M. Habib, Deep learning based semantic similarity detection using text data. Inf. Technol. Control 49 (2020)

    Google Scholar 

  27. I. Markov, H. Gómez-Adorno, J.P. Posadas-Durán, G. Sidorov, A. Gelbukh, Author profiling with doc2vec neural network-based document embeddings, in Advances in Soft Computing. ed. by O. Pichardo-Lagunas, S. Miranda-Jiménez (Springer International Publishing, Cham, 2017), pp.117–131

    Chapter  Google Scholar 

  28. A.M. Memon, M.E. Pollack, M.L. Soffa, A planning-based approach to GUI testing, in Proceedings of The 13th International Software/Internet Quality Week (2000)

    Google Scholar 

  29. R. Ramler, M. Felderer, Requirements for integrating defect prediction and risk-based testing, in 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (IEEE, 2016), pp. 359–362

    Google Scholar 

  30. S. Tahvili, W. Afzal, M. Saadatmand, M. Bohlin, S.H. Ameerjan, Espret: a tool for execution time estimation of manual test cases. J. Syst. Softw. 161, 1–43 (2018)

    Google Scholar 

  31. S. Tahvili, M. Bohlin, M. Saadatmand, S. Larsson, W. Afzal, D. Sundmark, Cost-benefit analysis of using dependency knowledge at integration testing, in The 17th International Conference On Product-Focused Software Process Improvement (2016)

    Google Scholar 

  32. S. Tahvili, L. Hatvani, M. Felderer, W. Afzal, M. Bohlin, Automated functional dependency detection between test cases using doc2vec and clustering, in 2019 IEEE International Conference On Artificial Intelligence Testing (AITest) (IEEE, 2019), pp. 19–26

    Google Scholar 

  33. S. Tahvili, L. Hatvani, M. Felderer, W. Afzal, M. Saadatmand, M. Bohlin, Cluster-based test scheduling strategies using semantic relationships between test specifications, in Proceedings of the 5th International Workshop on Requirements Engineering and Testing (2018), pp. 1–4

    Google Scholar 

  34. S. Tahvili, L. Hatvani, E. Ramentol, R. Pimentel, W. Afzal, F. Herrera, A novel methodology to classify test cases using natural language processing and imbalanced learning. Eng. Appl. Artif. Intell. 95, 1–13 (2020)

    Article  Google Scholar 

  35. J. Tang, Towards automation in software test life cycle based on multi-agent, in International Conference on Computational Intelligence and Software Engineering (IEEE, 2010), pp. 1–4

    Google Scholar 

  36. M. Utting, A. Pretschner, B. Legeard, A taxonomy of model-based testing approaches. Softw. Test., Verif. Reliab. 22(5), 297–312 (2012)

    Article  Google Scholar 

  37. M. Winikoff, Future directions for agent-based software engineering. IJAOSE 3(4), 402–410 (2009)

    Article  Google Scholar 

  38. M. Wooldridge, Agent-based software engineering. IEE Proc.-Softw. 144(1), 26–37 (1997)

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by the Austrian Science Fund (FWF): I 4701-N and the project ConTest funded by the Austrian Research Promotion Agency (FFG). Eduard Enoiu was partially supported by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 957212.

Author information

Authors and Affiliations

  1. University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  2. Blekinge Institute of Technology, Karlskrona, Sweden

    Michael Felderer

  3. Mälardalen University, Vësterås, Sweden

    Eduard Paul Enoiu & Sahar Tahvili

  4. Ericsson AB, Stockholm, Sweden

    Sahar Tahvili

Authors
  1. Michael Felderer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eduard Paul Enoiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sahar Tahvili
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Felderer .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Numerical Analysis, University of Córdoba, Córdoba, Córdoba, Spain

    José Raúl Romero

  2. Department of Computer Science and Engineering, University of Cádiz, Puerto Real, Cádiz, Spain

    Inmaculada Medina-Bulo

  3. ITIS Software, Department of Languages and Computing Sciences, University of Malaga, Málaga, Málaga, Spain

    Francisco Chicano

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Felderer, M., Enoiu, E.P., Tahvili, S. (2023). Artificial Intelligence Techniques in System Testing. In: Romero, J.R., Medina-Bulo, I., Chicano, F. (eds) Optimising the Software Development Process with Artificial Intelligence. Natural Computing Series. Springer, Singapore. https://doi.org/10.1007/978-981-19-9948-2_8

Download citation

  • DOI: https://doi.org/10.1007/978-981-19-9948-2_8

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-19-9947-5

  • Online ISBN: 978-981-19-9948-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation