Cuong Nguyen
ABSTRACT
As more software systems publish and use web services or APIs today, automated API testing is an important activity to help effectively ensure the quality of software services before they are released for their usage. Generating test scripts and data is a crucial step to perform API test automation successfully. In this paper, we propose an approach leveraging GPT, a large language model, and API’s Swagger specification to automatically generate test scripts and test data for API testing. Our approach also applies GPT’s self-refining with the feedback by executing tests on Katalon. We evaluate our proposed approach using a data set of seven APIs consisting of 157 endpoints and 179 operations. The result shows that while our approach generates fewer test scripts and data inputs, it can cover more successful status codes of 2xx than a state-of-the-art tool. This result suggests that leveraging the ability of GPT as a large language model to interpret API’s Swagger specification has the potential for improving the efficacy of generating test scripts and data for API testing.
- Juan C Alonso. 2021. Automated generation of realistic test inputs for web APIs. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1666–1668.Google Scholar
Digital Library
- Fuad Sameh Alshraiedeh and Norliza Katuk. 2021. A URI parsing technique and algorithm for anti-pattern detection in RESTful Web services. International Journal of Web Information Systems 17, 1 (2021), 1–17.Google ScholarCross Ref
- Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM) 28, 1 (2019), 1–37.Google ScholarDigital Library
- Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. Restler: Stateful rest api fuzzing. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 748–758.Google ScholarDigital Library
- Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K Lahiri. 2022. Toga: A neural method for test oracle generation. In Proceedings of the 44th International Conference on Software Engineering. 2130–2141.Google ScholarDigital Library
- Hamza Ed-Douibi, Javier Luis Cánovas Izquierdo, and Jordi Cabot. 2018. Automatic generation of test cases for REST APIs: A specification-based approach. In 2018 IEEE 22nd international enterprise distributed object computing conference (EDOC). IEEE, 181–190.Google Scholar
- Patrice Godefroid, Bo-Yuan Huang, and Marina Polishchuk. 2020. Intelligent REST API data fuzzing. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 725–736.Google ScholarDigital Library
- Zac Hatfield-Dodds and Dmitry Dygalo. 2022. Deriving semantics-aware fuzzers from web api schemas. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 345–346.Google ScholarDigital Library
- Stefan Karlsson, Adnan Čaušević, and Daniel Sundmark. 2020. QuickREST: Property-based test generation of OpenAPI-described RESTful APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). IEEE, 131–141.Google ScholarCross Ref
- Myeongsoo Kim, Davide Corradini, Saurabh Sinha, Alessandro Orso, Michele Pasqua, Rachel Tzoref-Brill, and Mariano Ceccato. 2023. Enhancing REST API Testing with NLP Techniques. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1232–1243.Google ScholarDigital Library
- Nuno Laranjeiro, João Agnelo, and Jorge Bernardino. 2021. A black box tool for robustness testing of REST services. IEEE Access 9 (2021), 24738–24754.Google ScholarCross Ref
- Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. 2022. Morest: model-based RESTful API testing with execution feedback. In Proceedings of the 44th International Conference on Software Engineering. 1406–1417.Google ScholarDigital Library
- Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 (2023).Google Scholar
- Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2020. RESTest: Black-box constraint-based testing of RESTful web APIs. In Service-Oriented Computing: 18th International Conference, ICSOC 2020, Dubai, United Arab Emirates, December 14–17, 2020, Proceedings 18. Springer, 459–475.Google ScholarDigital Library
- OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023).Google Scholar
- Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. Resttestgen: automated black-box testing of restful apis. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). IEEE, 142–152.Google ScholarCross Ref
- Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial testing of restful apis. In Proceedings of the 44th International Conference on Software Engineering. 426–437.Google ScholarDigital Library
- Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial testing of restful apis. In Proceedings of the 44th International Conference on Software Engineering. 426–437.Google ScholarDigital Library
- Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. 1–10.Google ScholarDigital Library
- Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601 (2023).Google Scholar
- Man Zhang and Andrea Arcuri. 2021. Adaptive hypermutation for search-based system test generation: A study on rest apis with evomaster. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021), 1–52.Google Scholar
Index Terms
- An Approach to Generating API Test Scripts Using GPT
Recommendations
Automated test generation for REST APIs: no time to rest yet
ISSTA 2022: Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and AnalysisModern web services routinely provide REST APIs for clients to access their functionality. These APIs present unique challenges and opportunities for automated testing, driving the recent development of many techniques and tools that generate test cases ...
A Formal Approach to Secure Design of RESTful Web APIs Using SOFL
Structured Object-Oriented Formal Language and MethodAbstractA primary concern in the design and development of a RESTful Application Programming Interfaces (APIs) is API security. A RESTful API provides data over the network using HTTP and must not violate any of its security properties. When APIs are ...
Generating API Test Data Using Deep Reinforcement Learning
ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering WorkshopsTesting is critical to ensure the quality of widely-used web APIs. Automatic test data generation can help to reduce cost and improve overall effectiveness. This is commonly accomplished by using the powerful concept of search-based software testing (...
Comments