ABSTRACT
RESTful services are commonly documented using OpenAPI specifications. Although numerous automated testing techniques have been proposed that leverage the machine-readable part of these specifications to guide test generation, their human-readable part has been mostly neglected. This is a missed opportunity, as natural language descriptions in the specifications often contain relevant information, including example values and inter-parameter dependencies, that can be used to improve test generation. In this spirit, we propose NLPtoREST, an automated approach that applies natural language processing techniques to assist REST API testing. Given an API and its specification, NLPtoREST extracts additional OpenAPI rules from the human-readable part of the specification. It then enhances the original specification by adding these rules to it. Testing tools can transparently use the enhanced specification to perform better test case generation. Because rule extraction can be inaccurate, due to either the intrinsic ambiguity of natural language or mismatches between documentation and implementation, NLPtoREST also incorporates a validation step aimed at eliminating spurious rules. We performed studies to assess the effectiveness of our rule extraction and validation approach, and the impact of enhanced specifications on the performance of eight state-of-the-art REST API testing tools. Our results are encouraging and show that NLPtoREST can extract many relevant rules with high accuracy, which can in turn significantly improve testing tools’ performance.
- J. C. Alonso, A. Martin-Lopez, S. Segura, J. Garcia, and A. Ruiz-Cortes. 2023. ARTE: Automated Generation of Realistic Test Inputs for Web APIs. IEEE Transactions on Software Engineering, 49, 01 (2023), jan, 348–363. issn:1939-3520 https://doi.org/10.1109/TSE.2022.3150618 Google ScholarCross Ref
- Andrea Arcuri. 2019. RESTful API Automated Test Case Generation with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM), 28, 1 (2019), Article 3, jan, 37 pages. issn:1049-331X https://doi.org/10.1145/3293455 Google ScholarDigital Library
- Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. In Proceedings of the 41st International Conference on Software Engineering (ICSE ’19). IEEE Press, Piscataway, NJ, USA. 748–758. https://doi.org/10.1109/ICSE.2019.00083 Google ScholarDigital Library
- Mourad Badri, Linda Badri, and Marius Naha. 2004. A Use Case Driven Testing Process: Towards a Formal Approach Based on UML Collaboration Diagrams. In Formal Approaches to Software Testing, Alexandre Petrenko and Andreas Ulrich (Eds.). Springer Berlin Heidelberg, Berlin, Germany. 223–235. isbn:978-3-540-24617-6 https://doi.org/10.1007/978-3-540-24617-6_16 Google ScholarCross Ref
- Christian Bizer, Jens Lehmann, Georgi Kobilarov, Sören Auer, Christian Becker, Richard Cyganiak, and Sebastian Hellmann. 2009. Dbpedia-a crystallization point for the web of data. Journal of web semantics, 7, 3 (2009), 154–165. Google ScholarDigital Library
- Arianna Blasi, Alberto Goffi, Konstantin Kuznetsov, Alessandra Gorla, Michael D. Ernst, Mauro Pezze, and Sergio Delgado Castellanos. 2018. Translating Code Comments to Procedure Specifications. In Proceedings of the 2018 International Symposium on Software Testing and Analysis (ISSTA 2018) (ISSTA ’18). Association for Computing Machinery, New York, NY, USA. 242–253. isbn:9781450356992 https://doi.org/10.1145/3213846.3213872 Google ScholarDigital Library
- Arianna Blasi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezze. 2022. Call Me Maybe: Using NLP to Automatically Generate Unit Test Cases Respecting Temporal Constraints. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE ’22). Association for Computing Machinery, New York, NY, USA. Article 19, 11 pages. isbn:9781450394758 https://doi.org/10.1145/3551349.3556961 Google ScholarDigital Library
- Hanyang Cao, Jean-Rémy Falleri, and Xavier Blanc. 2017. Automated Generation of REST API Specification from Plain HTML Documentation. In Service-Oriented Computing, Michael Maximilien, Antonio Vallecillo, Jianmin Wang, and Marc Oriol (Eds.). Springer International Publishing, New York, NY, USA. 453–461. isbn:978-3-319-69035-3 https://doi.org/10.1007/978-3-319-69035-3_32 Google ScholarDigital Library
- Davide Corradini, Amedeo Zampieri, Michele Pasqua, Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2022. Automated black-box testing of nominal and error scenarios in RESTful APIs. Software Testing, Verification and Reliability, 32 (2022), 01, https://doi.org/10.1002/stvr.1808 Google ScholarCross Ref
- Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-Based Software Architectures. Ph. D. Dissertation. University of California, Irvine. Google Scholar
- The Linux Foundation. 2022. OpenAPI specification. https://spec.openapis.org/oas/v3.1.0 Google Scholar
- Yoav Goldberg and Omer Levy. 2014. word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arxiv:1402.3722. Google Scholar
- Google. 2023. Google Bard. https://bard.google.com/ Google Scholar
- Nitin Hardeniya, Jacob Perkins, Deepti Chopra, Nisheeth Joshi, and Iti Mathur. 2016. Natural language processing: python and NLTK. Packt Publishing Ltd, Birmingham, UK. Google Scholar
- Zac Hatfield-Dodds and Dmitry Dygalo. 2022. Deriving Semantics-Aware Fuzzers from Web API Schemas. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 345–346. isbn:9781450392235 https://doi.org/10.1145/3510454.3528637 Google ScholarDigital Library
- Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. FastText.zip: Compressing text classification models. arxiv:1612.03651. Google Scholar
- Daniel Jurafsky and James H. Martin. 2021. Speech and Language Processing: Constituency Parsing. https://web.stanford.edu/ jurafsky/slp3/13.pdf Google Scholar
- Myeongsoo Kim, Qi Xin, Saurabh Sinha, and Alessandro Orso. ’22. Automated test generation for REST APIs: no time to rest yet. In ISSTA ’22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, South Korea, July 18 - 22, 2022, Sukyoung Ryu and Yannis Smaragdakis (Eds.). ACM, New York, NY, USA. 289–301. https://doi.org/10.1145/3533767.3534401 Google ScholarDigital Library
- Kerry Kimbrough. 2023. Tcases. https://github.com/Cornutum/tcases Google Scholar
- Dan Klein and Christopher D. Manning. 2003. Accurate Unlexicalized Parsing. In Proceedings of the 41st annual meeting of the association for computational linguistics. Association for Computational Linguistics, Edinburgh, Scotland. 423–430. Google Scholar
- LanguageTool. 2023. LanguageTool REST API. https://languagetool.org/proofreading-api Google Scholar
- Nuno Laranjeiro, João Agnelo, and Jorge Bernardino. 2021. A Black Box Tool for Robustness Testing of REST Services. IEEE Access, 9 (2021), 24738–24754. https://doi.org/10.1109/ACCESS.2021.3056505 Google ScholarCross Ref
- Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. 2022. Morest: Model-Based RESTful API Testing with Execution Feedback. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 1406–1417. isbn:9781450392211 https://doi.org/10.1145/3510003.3510133 Google ScholarDigital Library
- Bogdan Marculescu, Man Zhang, and Andrea Arcuri. 2022. On the Faults Found in REST APIs by Automated Test Generation. ACM Trans. Softw. Eng. Methodol., 31, 3 (2022), Article 41, mar, 43 pages. https://doi.org/10.1145/3491038 Google ScholarDigital Library
- Alberto Martin-Lopez, Sergio Segura, Carlos Müller, and Antonio Ruiz-Cortés. 2022. Specification and Automated Analysis of Inter-Parameter Dependencies in Web APIs. IEEE Transactions on Services Computing, 15, 4 (2022), 2342–2355. https://doi.org/10.1109/TSC.2021.3050610 Google ScholarCross Ref
- Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. A Catalogue of Inter-Parameter Dependencies in RESTful Web APIs. In Service-Oriented Computing: 17th International Conference, ICSOC 2019, Toulouse, France, October 28–31, 2019, Proceedings. Springer-Verlag, Berlin, Heidelberg. 399–414. isbn:978-3-030-33701-8 https://doi.org/10.1007/978-3-030-33702-5_31 Google ScholarDigital Library
- Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2021. RESTest: Automated Black-Box Testing of RESTful Web APIs. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021). Association for Computing Machinery, New York, NY, USA. 682–685. isbn:9781450384599 https://doi.org/10.1145/3460319.3469082 Google ScholarDigital Library
- Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2022. Online Testing of RESTful APIs: Promises and Challenges. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ’22). Association for Computing Machinery, New York, NY, USA. 408–420. isbn:9781450394130 https://doi.org/10.1145/3540250.3549144 Google ScholarDigital Library
- Manish Motwani and Yuriy Brun. 2019. Automatically Generating Precise Oracles from Structured Natural Language Specifications. In Proceedings of the 41st International Conference on Software Engineering (ICSE ’19). IEEE Press, Piscataway, NJ, USA. 188–199. https://doi.org/10.1109/ICSE.2019.00035 Google ScholarDigital Library
- Myeongsoo Kim and Davide Corradini. 2023. Experiment infrastructure and data for NLPtoREST. https://github.com/codingsoo/nlp2rest Google Scholar
- OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774. Google Scholar
- OpenAPI. 2023. OpenAPI standard. https://www.openapis.org Google Scholar
- Johannes Ryser and Martin Glinz. 1999. A scenario-based approach to validating and testing software systems using statecharts. https://doi.org/10.5167/uzh-205008 Google ScholarCross Ref
- SmartBear Software. 2023. OpenAPI Extensions. https://swagger.io/docs/specification/openapi-extensions/ Google Scholar
- EclEmma Team. 2023. JaCoCo. https://www.eclemma.org/jacoco/ Google Scholar
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arxiv:2302.13971. Google Scholar
- Chunhui Wang, Fabrizio Pastore, and Lionel Briand. 2018. Automated Generation of Constraints from Use Case Specifications to Support System Testing. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). IEEE, Piscataway, NJ, USA. 23–33. https://doi.org/10.1109/ICST.2018.00013 Google ScholarCross Ref
- Chunhui Wang, Fabrizio Pastore, Arda Goknil, and Lionel C. Briand. 2022. Automatic Generation of Acceptance Test Cases From Use Case Specifications: An NLP-Based Approach. IEEE Transactions on Software Engineering, 48, 2 (2022), 585–616. https://doi.org/10.1109/TSE.2020.2998503 Google ScholarCross Ref
- Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial Testing of RESTful APIs. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 426–437. isbn:9781450392211 https://doi.org/10.1145/3510003.3510151 Google ScholarDigital Library
- Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial Testing of RESTful APIs. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 426–437. isbn:9781450392211 https://doi.org/10.1145/3510003.3510151 Google ScholarDigital Library
- Jinqiu Yang, Erik Wittern, Annie T. T. Ying, Julian Dolby, and Lin Tan. 2018. Towards Extracting Web API Specifications from Documentation. In Proceedings of the 15th International Conference on Mining Software Repositories (MSR ’18). Association for Computing Machinery, New York, NY, USA. 454–464. isbn:9781450357166 https://doi.org/10.1145/3196398.3196411 Google ScholarDigital Library
Index Terms
- Enhancing REST API Testing with NLP Techniques
Recommendations
Generating REST API Specifications through Static Analysis
ICSE '24: Proceedings of the IEEE/ACM 46th International Conference on Software EngineeringWeb Application Programming Interfaces (APIs) allow services to be accessed over the network. RESTful (or REST) APIs, which use the REpresentation State Transfer (REST) protocol, are a popular type of web API. To use or test REST APIs, developers use ...
Comments