skip to main content
10.1145/3460319.3469082acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

RESTest: automated black-box testing of RESTful web APIs

Published: 11 July 2021 Publication History

Abstract

Testing RESTful APIs thoroughly is critical due to their key role in software integration. Existing tools for the automated generation of test cases in this domain have shown great promise, but their applicability is limited as they mostly rely on random inputs, i.e., fuzzing. In this paper, we present RESTest, an open source black-box testing framework for RESTful web APIs. Based on the API specification, RESTest supports the generation of test cases using different testing techniques such as fuzzing and constraint-based testing, among others. RESTest is developed as a framework and can be easily extended with new test case generators and test writers for different programming languages. We evaluate the tool in two scenarios: offline and online testing. In the former, we show how RESTest can efficiently generate realistic test cases (test inputs and test oracles) that uncover bugs in real-world APIs. In the latter, we show RESTest's capabilities as a continuous testing and monitoring framework. Demo video: https://youtu.be/1f_tjdkaCKo.

References

[1]
[n.d.]. Allure - Test report framework. http://allure.qatools.ru/ accessed April 2021.
[2]
[n.d.]. API Fortress. https://apifortress.com/ accessed April 2021.
[3]
[n.d.]. DBpedia. https://www.dbpedia.org/ accessed April 2021.
[4]
[n.d.]. OpenAPI Specification. https://www.openapis.org accessed April 2021.
[5]
[n.d.]. PayPal API. https://developer.paypal.com/docs/api/ accessed April 2021.
[6]
[n.d.]. Postman. https://www.getpostman.com accessed April 2021.
[7]
[n.d.]. ProgrammableWeb API Directory. http://www.programmableweb.com/ accessed April 2021.
[8]
[n.d.]. ReadyAPI. https://smartbear.com/product/ready-api/overview/ accessed April 2021.
[9]
[n.d.]. REST Assured. http://rest-assured.io accessed April 2021.
[10]
[n.d.]. YouTube Data API. https://developers.google.com/youtube/v3/ accessed April 2021.
[11]
Andrea Arcuri. 2019. RESTful API Automated Test Case Generation with EvoMaster. ACM TOSEM, 28, 1 (2019), 1–37.
[12]
Andrea Arcuri. 2021. Automated Blackbox and Whitebox Testing of RESTful APIs With EvoMaster. IEEE Software.
[13]
Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. In ICSE. 748–758.
[14]
T. Y. Chen, H. Leung, and I. K. Mak. 2005. Adaptive Random Testing. In ASIAN. 320–329.
[15]
Hamza Ed-douibi, Javier Luis Cánovas Izquierdo, and Jordi Cabot. 2018. Automatic Generation of Test Cases for REST APIs: A Specification-Based Approach. In EDOC. 181–190.
[16]
Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-based Software Architectures. Ph.D. Dissertation. isbn:0-599-87118-0
[17]
Antonio Gamez-Diaz, Pablo Fernandez, and Antonio Ruiz-Cortes. 2019. Automating SLA-Driven API Development with SLA4OAI. In ICSOC. 20–35.
[18]
Stefan Karlsson, Adnan Causevic, and Daniel Sundmark. 2020. QuickREST: Property-based Test Generation of OpenAPI Described RESTful APIs. In ICST.
[19]
Alberto Martin-Lopez, Sergio Segura, Carlos Müller, and Antonio Ruiz-Cortés. 2020. Specification and Automated Analysis of Inter-Parameter Dependencies in Web APIs. IEEE TSC.
[20]
Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. A Catalogue of Inter-Parameter Dependencies in RESTful Web APIs. In ICSOC. 399–414.
[21]
Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. Test Coverage Criteria for RESTful Web APIs. In A-TEST. 15–21.
[22]
Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2020. RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs. In ICSOC.
[23]
Leonard Richardson, Mike Amundsen, and Sam Ruby. 2013. RESTful Web APIs. O’Reilly Media, Inc. isbn:1449358063, 9781449358068
[24]
Sergio Segura, Amador Durán, Javier Troya, and Antonio Ruiz-Cortés. 2019. Metamorphic Relation Patterns for Query-Based Systems. In MET. 24–31.
[25]
Sergio Segura, José A Parejo, Javier Troya, and Antonio Ruiz-Cortés. 2018. Metamorphic Testing of RESTful Web APIs. IEEE TSE, 44, 11 (2018), 1083–1099.
[26]
Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RestTestGen: Automated Black-Box Testing of RESTful APIs. In ICST.

Cited By

View all
  • (2025)Fuzzing frameworks for server-side web applications: a surveyInternational Journal of Information Security10.1007/s10207-024-00979-w24:2Online publication date: 5-Feb-2025
  • (2024)Beyond REST: Introducing APIF for Comprehensive API Vulnerability FuzzingProceedings of the 27th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3678890.3678928(435-449)Online publication date: 30-Sep-2024
  • (2024)Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIsACM Transactions on Software Engineering and Methodology10.1145/365215733:6(1-36)Online publication date: 27-Jun-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis
July 2021
685 pages
ISBN:9781450384599
DOI:10.1145/3460319
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. REST
  2. black-box testing
  3. web APIs

Qualifiers

  • Research-article

Funding Sources

  • Junta de Andalucía (Consejería de Economía y Conocimiento)
  • Ministerio de Educación y Formación Profesional
  • Ministerio de Ciencia, Innovación y Universidades

Conference

ISSTA '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)223
  • Downloads (Last 6 weeks)11
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Fuzzing frameworks for server-side web applications: a surveyInternational Journal of Information Security10.1007/s10207-024-00979-w24:2Online publication date: 5-Feb-2025
  • (2024)Beyond REST: Introducing APIF for Comprehensive API Vulnerability FuzzingProceedings of the 27th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3678890.3678928(435-449)Online publication date: 30-Sep-2024
  • (2024)Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIsACM Transactions on Software Engineering and Methodology10.1145/365215733:6(1-36)Online publication date: 27-Jun-2024
  • (2024)Leveraging Large Language Models to Improve REST API TestingProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results10.1145/3639476.3639769(37-41)Online publication date: 14-Apr-2024
  • (2024)ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite GenerationIEEE Transactions on Software Engineering10.1109/TSE.2024.338236550:6(1340-1359)Online publication date: 29-Mar-2024
  • (2024)Coverage Goal Selector for Combining Multiple Criteria in Search-Based Unit Test GenerationIEEE Transactions on Software Engineering10.1109/TSE.2024.336661350:4(854-883)Online publication date: 16-Feb-2024
  • (2024)KAT: Dependency-Aware Automated API Testing with Large Language Models2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00017(82-92)Online publication date: 27-May-2024
  • (2024)Developing a real-time detection and visualization of landslide hazards using web-GIS: A case study in Pacet, Mojokerto, East Java, IndonesiaINTERNATIONAL CONFERENCE ON ENVIRONMENTAL, MINING, AND SUSTAINABLE DEVELOPMENT 202210.1063/5.0184132(050001)Online publication date: 2024
  • (2024)Exploring behaviours of RESTful APIs in an industrial settingSoftware Quality Journal10.1007/s11219-024-09686-032:3(1287-1324)Online publication date: 1-Sep-2024
  • (2024)Tool report: EvoMaster—black and white box search-based fuzzing for REST, GraphQL and RPC APIsAutomated Software Engineering10.1007/s10515-024-00478-132:1Online publication date: 29-Nov-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media