Huayao Wu
Changhai Nie
ABSTRACT
This paper presents RestCT, a systematic and fully automatic approach that adopts Combinatorial Testing (CT) to test RESTful APIs. RestCT is systematic in that it covers and tests not only the interactions of a certain number of operations in RESTful APIs, but also the interactions of particular input-parameters in every single operation. This is realised by a novel two-phase test case generation approach, which first generates a constrained sequence covering array to determine the execution orders of available operations, and then applies an adaptive strategy to generate and refine several constrained covering arrays to concretise input-parameters of each operation. RestCT is also automatic in that its application relies on only a given Swagger specification of RESTful APIs. The creation of CT test models (especially, the inferring of dependency relationships in both operations and input-parameters), and the generation and execution of test cases are performed without any human intervention. Experimental results on 11 real-world RESTful APIs demonstrate the effectiveness and efficiency of RestCT. In particular, RestCT can find eight new bugs, where only one of them can be triggered by the state-of-the-art testing tool of RESTful APIs.
- Juan C Alonso. 2021. Automated generation of realistic test inputs for web APIs. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 1666--1668. Google Scholar
Digital Library
- Fuad Sameh Alshraiedeh and Norliza Katuk. 2020. A URI parsing technique and algorithm for anti-pattern detection in RESTful Web services. International Journal of Web Information Systems 17, 1 (2020), 1--17. Google ScholarCross Ref
- Andrea Arcuri. 2017. RESTful API automated test case generation. In International Conference on Software Quality, Reliability and Security (QRS). 9--20. Google ScholarCross Ref
- Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM) 28, 1 (2019), 1--37. Google ScholarDigital Library
- Andrea Arcuri. 2020. Automated blackbox and whitebox testing of RESTful APIs with EvoMaster. IEEE Software (2020). Google ScholarCross Ref
- Andrea Arcuri and Juan P Galeotti. 2020. Handling SQL databases in automated system test generation. ACM Transactions on Software Engineering and Methodology (TOSEM) 29, 4 (2020), 1--31. Google ScholarDigital Library
- Andrea Arcuri and Juan P Galeotti. 2021. Enhancing search-based testing with testability transformations for existing APIs. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021), 1--34. Google ScholarDigital Library
- Vaggelis Atlidakis, Roxana Geambasu, Patrice Godefroid, Marina Polishchuk, and Baishakhi Ray. 2020. Pythia: grammar-based fuzzing of REST APIs with coverage-guided feedback and learning-based mutations. arXiv preprint (2020). https://arxiv.org/abs/2005.11498.Google Scholar
- Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: stateful REST API fuzzing. In International Conference on Software Engineering (ICSE). 748--758. Google ScholarDigital Library
- Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2020. Checking security properties of cloud service REST APIs. In International Conference on Software Testing, Validation and Verification (ICST). 387--397. Google ScholarCross Ref
- Steven Bucaille, Javier Luis Cánovas Izquierdo, Hamza Ed-Douibi, and Jordi Cabot. 2020. An OpenAPI-based testing framework to monitor non-functional properties of REST APIs. In International Conference on Web Engineering. 533--537. Google ScholarDigital Library
- Sujit Kumar Chakrabarti and Prashant Kumar. 2009. Test-the-REST: An approach to testing RESTful web-services. In Computation World: Future Computing, Service Computation, Cognitive, Adaptive, Content, Patterns. 302--308. Google ScholarDigital Library
- Sujit Kumar Chakrabarti and Reswin Rodriquez. 2010. Connectedness testing of RESTful web-services. In India software engineering conference. 143--152. Google ScholarDigital Library
- Yixiong Chen, Yang Yang, Zhanyao Lei, Mingyuan Xia, and Zhengwei Qi. 2021. Bootstrapping automated testing for RESTful web services. In International Conference on Fundamental Approaches to Software Engineering. 46--66. Google ScholarDigital Library
- Davide Corradini, Amedeo Zampieri, Michele Pasqua, and Mariano Ceccato. 2021. Empirical comparison of black-box test case generation tools for RESTful APIs. In International Working Conference on Source Code Analysis and Manipulation (SCAM). 226--236. Google ScholarCross Ref
- Davide Corradini, Amedeo Zampieri, Michele Pasqua, and Mariano Ceccato. 2021. Restats: A test coverage tool for RESTful APIs. In International Conference on Software Maintenance and Evolution (ICSME). 594--598. Google ScholarCross Ref
- Alexandre L. Correa, Thiago Silva-de-Souza, Eber Assis Schmitz, and Antonio Juarez Alencar. 2012. Defining RESTful web services test cases from UML models. In International Conference on Software Engineering & Knowledge Engineering. 319--323.Google Scholar
- Hamza Ed-Douibi, Javier Luis Cánovas Izquierdo, and Jordi Cabot. 2018. Automatic generation of test cases for REST APIs: a specification-based approach. In International Enterprise Distributed Object Computing Conference. 181--190. Google ScholarCross Ref
- Tobias Fertig and Peter Braun. 2015. Model-driven testing of RESTful APIs. In International Conference on World Wide Web: Companion. 1497--1502. Google ScholarDigital Library
- Patrice Godefroid, Bo-Yuan Huang, and Marina Polishchuk. 2020. Intelligent REST API data fuzzing. In Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 725--736. Google ScholarDigital Library
- Patrice Godefroid, Daniel Lehmann, and Marina Polishchuk. 2020. Differential regression testing for REST APIs. In International Symposium on Software Testing and Analysis (ISSTA). 312--323. Google ScholarDigital Library
- Stefan Karlsson, Adnan Čaušević, and Daniel Sundmark. 2020. QuickREST: property-based test generation of OpenAPI-described RESTful APIs. In International Conference on Software Testing, Validation and Verification (ICST). 131--141. Google ScholarCross Ref
- D Richard Kuhn, James M Higdon, James F Lawrence, Raghu N Kacker, and Yu Lei. 2012. Combinatorial methods for event sequence testing. In International Conference on Software Testing, Verification and Validation (ICST). 601--609. Google ScholarCross Ref
- D. Richard Kuhn, Raghu N. Kacker, and Yu Lei. 2013. Introduction to combinatorial testing. CRC press.Google Scholar
- D. Richard Kuhn and Dolores R. Wallace. 2004. Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering (TSE) 30, 6 (2004), 418--421. Google ScholarDigital Library
- Pablo Lamela Seijas, Huiqing Li, and Simon Thompson. 2013. Towards property-based testing of RESTful web services. In ACM SIGPLAN workshop on Erlang. 77--78. Google ScholarDigital Library
- Nuno Laranjeiro, João Agnelo, and Jorge Bernardino. 2021. A black box tool for robustness testing of REST services. IEEE Access 9 (2021), 24738--24754. Google ScholarCross Ref
- Li Li and Wu Chou. 2015. Compatibility modeling and testing of REST API based on REST chart. In International Conference on Web Information Systems and Technologies. 194--202. Google ScholarCross Ref
- Alberto Martin-Lopez. 2020. Automated analysis of inter-parameter dependencies in web APIs. In International Conference on Software Engineering: Companion (ICSE-Companion). 140--142. Google ScholarDigital Library
- Alberto Martin-Lopez, Sergio Segura, Carlos Muller, and Antonio Ruiz-Cortés. 2021. Specification and automated analysis of inter-parameter dependencies in web APIs. IEEE Transactions on Services Computing (2021). to-be-published. Google ScholarCross Ref
- Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. A catalogue of inter-parameter dependencies in RESTful web APIs. In International Conference on Service-Oriented Computing (ICSOC). 399--414. Google ScholarDigital Library
- Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. Test coverage criteria for RESTful web APIs. In International Workshop on Automating TEST Case Design, Selection, and Evaluation. 15--21. Google ScholarDigital Library
- Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2020. RESTest: black-box constraint-based testing of RESTful web APIs. In International Conference on Service-Oriented Computing (ICSOC). 459--475. Google ScholarDigital Library
- Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A comparison of 10 sampling algorithms for configurable systems. In International Conference on Software Engineering (ICSE). 643--654. Google ScholarDigital Library
- A Giuliano Mirabella, Alberto Martin-Lopez, Sergio Segura, Luis Valencia-Cabrera, and Antonio Ruiz-Cortés. 2021. Deep learning-based prediction of test input validity for RESTful APIs. In International Workshop on Deep Learning for Testing and Testing for Deep Learning. 9--16. Google ScholarCross Ref
- Changhai Nie and Hareton Leung. 2011. A survey of combinatorial testing. Comput. Surveys 43, 2 (2011), 1--29. Google ScholarDigital Library
- Justyna Petke, Myra B Cohen, Mark Harman, and Shin Yoo. 2015. Practical combinatorial interaction testing: Empirical findings on efficiency and early fault detection. IEEE Transactions on Software Engineering (TSE) 41, 9 (2015), 901--924. Google ScholarDigital Library
- Pedro Victor Pontes Pinheiro, Andre Takeshi Endo, and Adenilso Simao. 2013. Model-based testing of RESTful web services using UML protocol state machines. In Brazilian Workshop on Systematic and Automated Software Testing. 1--10.Google Scholar
- Leonard Richardson, Mike Amundsen, Michael Amundsen, and Sam Ruby. 2013. RESTful web APIs: services for a changing world. O'Reilly Media, Inc.Google Scholar
- Omur Sahin and Bahriye Akay. 2021. A discrete dynamic artificial bee colony with hyper-scout for RESTful web service API test suite generation. Applied Soft Computing 104 (2021), 107246. Google ScholarDigital Library
- Sergio Segura, José A Parejo, Javier Troya, and Antonio Ruiz-Cortés. 2018. Metamorphic testing of RESTful web APIs. IEEE Transactions on Software Engineering 44, 11 (2018), 1083--1099. Google ScholarCross Ref
- Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RestTestGen: automated black-box testing of RESTful APIs. In International Conference on Software Testing, Validation and Verification (ICST). 142--152. Google ScholarCross Ref
- Diba Vosta. 2020. Evaluation of the t-wise approach for testing REST APIs. Master's thesis. KTH, School of Electrical Engineering and Computer Science.Google Scholar
- Henry Vu, Tobias Fertig, and Peter Braun. 2018. Automation of integration testing of RESTful hypermedia systems: A model-driven approach. In International Conference on Web Information Systems and Technologies. 404--411. Google ScholarCross Ref
- Huayao Wu, Changhai Nie, Justyna Petke, Yue Jia, and Mark Harman. 2021. Comparative analysis of constraint handling techniques for constrained combinatorial testing. IEEE Transactions on Software Engineering (TSE) 47, 11 (2021), 2549--2562. Google ScholarCross Ref
- Man Zhang and Andrea Arcuri. 2021. Adaptive hypermutation for search-based system test generation: A study on REST APIs with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021), 1--52. Google ScholarDigital Library
- Man Zhang and Andrea Arcuri. 2021. Enhancing resource-based test case generation for RESTful APIs with SQL handling. In International Symposium on Search Based Software Engineering (SSBSE). 103--117. Google ScholarDigital Library
- Man Zhang, Bogdan Marculescu, and Andrea Arcuri. 2019. Resource-based test case generation for RESTful web services. In Genetic and Evolutionary Computation Conference (GECCO). 1426--1434. Google ScholarDigital Library
- Man Zhang, Bogdan Marculescu, and Andrea Arcuri. 2021. Resource and dependency based test case generation for RESTful Web services. Empirical Software Engineering 26, 4 (2021), 1--61. Google ScholarDigital Library
Index Terms
- Combinatorial testing of RESTful APIs
Recommendations
Online testing of RESTful APIs: promises and challenges
ESEC/FSE 2022: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software EngineeringOnline testing of web APIs—testing APIs in production—is gaining traction in industry. Platforms such as RapidAPI and Sauce Labs provide online testing and monitoring services of web APIs 24/7, typically by re-executing manually designed test cases on ...
RESTest: automated black-box testing of RESTful web APIs
ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and AnalysisTesting RESTful APIs thoroughly is critical due to their key role in software integration. Existing tools for the automated generation of test cases in this domain have shown great promise, but their applicability is limited as they mostly rely on ...
Testing RESTful APIs: A Survey
In industry, RESTful APIs are widely used to build modern Cloud Applications. Testing them is challenging, because not only do they rely on network communications, but also they deal with external services like databases. Therefore, there has been a large ...
Comments