skip to main content
10.1145/3597926.3598114acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

AGORA: Automated Generation of Test Oracles for REST APIs

Published: 13 July 2023 Publication History

Abstract

Test case generation tools for REST APIs have grown in number and complexity in recent years. However, their advanced capabilities for automated input generation contrast with the simplicity of their test oracles, which limit the types of failures they can detect to crashes, regressions, and violations of the API specification or design best practices. In this paper, we present AGORA, an approach for the automated generation of test oracles for REST APIs through the detection of invariants—properties of the output that should always hold. In practice, AGORA aims to learn the expected behavior of an API by analyzing previous API requests and their corresponding responses. For this, we extended the Daikon tool for dynamic detection of likely invariants, including the definition of new types of invariants and the implementation of an instrumenter called Beet. Beet converts any OpenAPI specification and a collection of API requests and responses to a format processable by Daikon. As a result, AGORA currently supports the detection of up to 105 different types of invariants in REST APIs. AGORA achieved a total precision of 81.2% when tested on a dataset of 11 operations from 7 industrial APIs. More importantly, the test oracles generated by AGORA detected 6 out of every 10 errors systematically seeded in the outputs of the APIs under test. Additionally, AGORA revealed 11 bugs in APIs with millions of users: Amadeus, GitHub, Marvel, OMDb and YouTube. Our reports have guided developers in improving their APIs, including bug fixes and documentation updates in GitHub. Since it operates in black-box mode, AGORA can be seamlessly integrated into existing API testing tools.

References

[1]
2022. DAIKON instrumenters. https://plse.cs.washington.edu/daikon/download/doc/daikon.html#Front-ends-_0028instrumentation_0029 Accessed September 2022
[2]
2022. OpenAPI Specification. https://www.openapis.org Accessed September 2022
[3]
2022. RapidAPI API directory. https://rapidapi.com/marketplace Accessed November 2022
[4]
2022. Spotify Web API. https://developer.spotify.com/web-api/ accessed September 2022
[5]
2023. Beet repository. https://github.com/isa-group/Beet
[6]
2023. JSONMutator. https://github.com/isa-group/JSONmutator Accessed January 2023
[7]
2023. Replication package. https://doi.org/10.5281/zenodo.7970822
[8]
2023. SRC Grand Finalists 2023. https://src.acm.org/grand-finalists/2023
[9]
2023. Visa Developer Center. https://developer.visa.com accessed January 2023
[10]
Afsoon Afzal, Claire Le Goues, and Christopher Steven Timperley. 2021. Mithra: Anomaly Detection as an Oracle for Cyberphysical Systems. IEEE Transactions on Software Engineering, 1–1. https://doi.org/10.1109/TSE.2021.3120680
[11]
Juan C. Alonso. 2022. Automated Generation of Test Oracles for RESTful APIs. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA. 1808–1810. isbn:9781450394130 https://doi.org/10.1145/3540250.3559080
[12]
Juan C. Alonso, Alberto Martin-Lopez, Sergio Segura, Jose Maria Garcia, and Antonio Ruiz-Cortes. 2022. ARTE: Automated Generation of Realistic Test Inputs for Web APIs. IEEE Transactions on Software Engineering, https://doi.org/10.1109/TSE.2022.3150618
[13]
2022. APIs.guru. https://apis.guru accessed October 2022
[14]
Andrea Arcuri. 2019. RESTful API Automated Test Case Generation with EvoMaster. ACM Transactions on Software Engineering and Methodology, 28, 1 (2019), 1–37. https://doi.org/10.1145/3293455
[15]
V. Atlidakis, P. Godefroid, and M. Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 748–758. https://doi.org/10.1109/ICSE.2019.00083
[16]
Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2020. Checking Security Properties of Cloud Services REST APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). 387–397. https://doi.org/10.1109/ICST46399.2020.00046
[17]
Jon Ayerdi, Valerio Terragni, Aitor Arrieta, Paolo Tonella, Goiuria Sagardui, and Maite Arratibel. 2021. Generating Metamorphic Relations for Cyber-Physical Systems with Genetic Programming: An Industrial Case Study. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2021). Association for Computing Machinery, New York, NY, USA. 1264–1274. isbn:9781450385626 https://doi.org/10.1145/3468264.3473920
[18]
Efe Barlas, Xin Du, and James C. Davis. 2022. Exploiting Input Sanitization for Regex Denial of Service. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 883–895. isbn:9781450392211 https://doi.org/10.1145/3510003.3510047
[19]
Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41, 5 (2015), 507–525. https://doi.org/10.1109/TSE.2014.2372785
[20]
Arianna Blasi, Alberto Goffi, Konstantin Kuznetsov, Alessandra Gorla, Michael D. Ernst, Mauro Pezzè, and Sergio Delgado Castellanos. 2018. Translating code comments to procedure specifications. In ISSTA 2018, Proceedings of the 2018 International Symposium on Software Testing and Analysis. Amsterdam, Netherlands. 242–253.
[21]
Arianna Blasi, Alessandra Gorla, Michael D. Ernst, Mauro Pezzè, and Antonio Carzaniga. 2021. MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation. Journal of Systems and Software, 181 (2021), Nov., 111041:1–13.
[22]
Houssem Ben Braiek and Foutse Khomh. 2020. On testing machine learning programs. Journal of Systems and Software, 164 (2020), 110542. issn:0164-1212 https://doi.org/10.1016/j.jss.2020.110542
[23]
Marcel Böhme, Charaka Geethal, and Van-Thuan Pham. 2020. Human-In-The-Loop Automatic Program Repair. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). 274–285. https://doi.org/10.1109/ICST46399.2020.00036
[24]
Tianyi Chen, Kihong Heo, and Mukund Raghothaman. 2021. Boosting Static Analysis Accuracy with Instrumented Test Executions. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2021). Association for Computing Machinery, New York, NY, USA. 1154–1165. isbn:9781450385626 https://doi.org/10.1145/3468264.3468626
[25]
Jake Cobb, James A. Jones, Gregory M. Kapfhammer, and Mary Jean Harrold. 2011. Dynamic Invariant Detection for Relational Databases. In Proceedings of the Ninth International Workshop on Dynamic Analysis (WODA ’11). Association for Computing Machinery, New York, NY, USA. 12–17. isbn:9781450308113 https://doi.org/10.1145/2002951.2002955
[26]
Davide Corradini, Michele Pasqua, and Mariano Ceccato. 2023. Automated Black-box Testing of Mass Assignment Vulnerabilities in RESTful APIs. https://doi.org/10.48550/ARXIV.2301.01261
[27]
Patrick Cousot and Radhia Cousot. 1992. Abstract Interpretation Frameworks. Journal of Logic and Computation, 2 (1992), 08, https://doi.org/10.1093/logcom/2.4.511
[28]
Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. 2022. TOGA: A Neural Method for Test Oracle Generation. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 2130–2141. isbn:9781450392211 https://doi.org/10.1145/3510003.3510141
[29]
Michael D. Ernst. 2003. Static and dynamic analysis: Synergy and duality. In WODA 2003: Workshop on Dynamic Analysis. Portland, OR, USA. 24–27.
[30]
Michael D. Ernst, Jake Cockrell, William G. Griswold, and David Notkin. 2001. Dynamically discovering likely program invariants to support program evolution. IEEE Transactions on Software Engineering, 27, 2 (2001), Feb., 99–123.
[31]
Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, and Chen Xiao. 2007. The Daikon system for dynamic detection of likely invariants. Science of Computer Programming, 69, 1 (2007), 35–45. issn:0167-6423 https://doi.org/10.1016/j.scico.2007.01.015 Special issue on Experimental Software and Toolkits
[32]
Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-based Software Architectures. Ph. D. Dissertation. University of California, Irvine. isbn:0-599-87118-0
[33]
Gordon Fraser and Andrea Arcuri. 2013. Whole Test Suite Generation. IEEE Transactions on Software Engineering, 39, 2 (2013), 276–291. https://doi.org/10.1109/TSE.2012.14
[34]
Gregory Gay, Sanjai Rayadurgam, and Mats P.E. Heimdahl. 2014. Improving the Accuracy of Oracle Verdicts through Automated Model Steering. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE ’14). Association for Computing Machinery, New York, NY, USA. 527–538. isbn:9781450330138 https://doi.org/10.1145/2642937.2642989
[35]
Luca Gazzola, Maayan Goldstein, Leonardo Mariani, Itai Segall, and Luca Ussi. 2020. Automatic Ex-Vivo Regression Testing of Microservices. In Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test (AST ’20). Association for Computing Machinery, New York, NY, USA. 11–20. isbn:9781450379571 https://doi.org/10.1145/3387903.3389309
[36]
2022. GitHub API. https://developer.github.com/v3/ accessed September 2022
[37]
C. Giuffrida, L. Cavallaro, and A. S. Tanenbaum. 2013. Practical automated vulnerability monitoring using program state invariants. In 2013 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). IEEE Computer Society, Los Alamitos, CA, USA. 1–12. issn:1530-0889 https://doi.org/10.1109/DSN.2013.6575318
[38]
Patrice Godefroid, Bo-Yuan Huang, and Marina Polishchuk. 2020. Intelligent REST API Data Fuzzing. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 725–736. isbn:9781450370431 https://doi.org/10.1145/3368089.3409719
[39]
Patrice Godefroid, Daniel Lehmann, and Marina Polishchuk. 2020. Differential Regression Testing for REST APIs. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2020). Association for Computing Machinery, New York, NY, USA. 312–323. isbn:9781450380089 https://doi.org/10.1145/3395363.3397374
[40]
Alberto Goffi, Alessandra Gorla, Michael D. Ernst, and Mauro Pezzè. 2016. Automatic generation of oracles for exceptional behaviors. In ISSTA 2016, Proceedings of the 2016 International Symposium on Software Testing and Analysis. Saarbrücken, Genmany. 213–224.
[41]
Amid Golmohammadi, Man Zhang, and Andrea Arcuri. 2022. Testing RESTful APIs: A Survey. https://doi.org/10.48550/ARXIV.2212.14604
[42]
Stewart Grant, Hendrik Cech, and Ivan Beschastnikh. 2018. Inferring and Asserting Distributed System Invariants. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18). Association for Computing Machinery, New York, NY, USA. 1149–1159. isbn:9781450356381 https://doi.org/10.1145/3180155.3180199
[43]
J. Haltermann and H. Wehrheim. 2022. Machine Learning Based Invariant Generation: A Framework and Reproducibility Study. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE Computer Society, Los Alamitos, CA, USA. 12–23. issn:2159-4848 https://doi.org/10.1109/ICST53961.2022.00012
[44]
Zac Hatfield-Dodds and Dmitry Dygalo. 2021. Deriving Semantics-Aware Fuzzers from Web API Schemas. arXiv preprint arXiv:2112.10328.
[45]
Rubing Huang, Weifeng Sun, Yinyin Xu, Haibo Chen, Dave Towey, and Xin Xia. 2021. A Survey on Adaptive Random Testing. IEEE Transactions on Software Engineering, 47, 10 (2021), 2052–2083. https://doi.org/10.1109/TSE.2019.2942921
[46]
Ali Reza Ibrahimzada, Yigit Varli, Dilara Tekinoglu, and Reyhaneh Jabbarvand. 2022. Perfect is the Enemy of Test Oracle. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA. 70–81. isbn:9781450394130 https://doi.org/10.1145/3540250.3549086
[47]
Daniel Jackson. 2012. Software Abstractions: Logic, Language, and Analysis. The MIT Press. isbn:0262017156
[48]
Daniel Jacobson, Greg Brail, and Dan Woods. 2011. APIs: A Strategy Guide. O’Reilly Media, Inc. isbn:1449308929, 9781449308926
[49]
Charaka Geethal Kapugama, Van-Thuan Pham, Aldeida Aleti, and Marcel Böhme. 2022. Human-in-the-Loop Oracle Learning for Semantic Bugs in String Processing Programs. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2022). Association for Computing Machinery, New York, NY, USA. 215–226. isbn:9781450393799 https://doi.org/10.1145/3533767.3534406
[50]
Stefan Karlsson, Adnan Causevic, and Daniel Sundmark. 2020. QuickREST: Property-based Test Generation of OpenAPI Described RESTful APIs. In International Conference on Software Testing, Validation and Verification. 131–141.
[51]
Myeongsoo Kim, Qi Xin, Saurabh Sinha, and Alessandro Orso. 2022. Automated Test Generation for REST APIs: No Time to Rest Yet. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2022). Association for Computing Machinery, New York, NY, USA. 289–301. isbn:9781450393799 https://doi.org/10.1145/3533767.3534401
[52]
Sumit Lahiri and Subhajit Roy. 2022. Almost Correct Invariants: Synthesizing Inductive Invariants by Fuzzing Proofs. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2022). Association for Computing Machinery, New York, NY, USA. 352–364. isbn:9781450393799 https://doi.org/10.1145/3533767.3534381
[53]
Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. 2022. Morest: Model-Based RESTful API Testing with Execution Feedback. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 1406–1417. isbn:9781450392211 https://doi.org/10.1145/3510003.3510133
[54]
R. Mahmood, J. Pennington, D. Tsang, T. Tran, and A. Bogle. 2022. A Framework for Automated API Fuzzing at Enterprise Scale. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE Computer Society, Los Alamitos, CA, USA. 377–388. issn:2159-4848 https://doi.org/10.1109/ICST53961.2022.00018
[55]
Alberto Martin-Lopez, Andrea Arcuri, Sergio Segura, and Antonio Ruiz-Cortés. 2021. Black-Box and White-Box Test Case Generation for RESTful APIs: Enemies or Allies? In International Symposium on Software Reliability Engineering.
[56]
Alberto Martin-Lopez, Sergio Segura, Carlos Müller, and Antonio Ruiz-Cortés. 2021. Specification and Automated Analysis of Inter-Parameter Dependencies in Web APIs. IEEE Transactions on Services Computing, https://doi.org/10.1109/TSC.2021.3050610
[57]
Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2020. RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs. In International Conference on Service-Oriented Computing. 459–475.
[58]
Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2021. RESTest: Automated Black-Box Testing of RESTful Web APIs. In International Symposium on Software Testing and Analysis.
[59]
Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2022. Online Testing of RESTful APIs: Promises and Challenges. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA. 408–420. isbn:9781450394130 https://doi.org/10.1145/3540250.3549144
[60]
Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2021. Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks. In Proceedings of the 43rd International Conference on Software Engineering (ICSE ’21). IEEE Press, 336–347. isbn:9781450390859 https://doi.org/10.1109/ICSE43902.2021.00041
[61]
Facundo Molina, Marcelo d’Amorim, and Nazareno Aguirre. 2022. Fuzzing Class Specifications. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 1008–1020. isbn:9781450392211 https://doi.org/10.1145/3510003.3510120
[62]
Facundo Molina, Renzo Degiovanni, Pablo Ponzio, Germán Regis, Nazareno Aguirre, and Marcelo Frias. 2019. Training Binary Classifiers as Data Structure Invariants. In Proceedings of the 41st International Conference on Software Engineering (ICSE ’19). IEEE Press, 759–770. https://doi.org/10.1109/ICSE.2019.00084
[63]
Facundo Molina, Pablo Ponzio, Nazareno Aguirre, and Marcelo Frias. 2021. EvoSpex: An Evolutionary Algorithm for Learning Postconditions. In Proceedings of the 43rd International Conference on Software Engineering (ICSE ’21). IEEE Press, 1223–1235. isbn:9781450390859 https://doi.org/10.1109/ICSE43902.2021.00112
[64]
Jeremy W. Nimmer and Michael D. Ernst. 2001. Static verification of dynamically detected program invariants: Integrating Daikon and ESC/Java. In RV 2001: Proceedings of the First Workshop on Runtime Verification. Paris, France.
[65]
Jeremy W. Nimmer and Michael D. Ernst. 2002. Automatic generation of program specifications. In ISSTA 2002, Proceedings of the 2002 International Symposium on Software Testing and Analysis. Rome, Italy. 232–242.
[66]
Manuel Palomo-Duarte, Antonio García-Domínguez, Inmaculada Medina-Bulo, Alejandro Alvarez-Ayllón, and Javier Santacruz. 2010. Takuan: A Tool for WS-BPEL Composition Testing Using Dynamic Invariant Generation. In Web Engineering, Boualem Benatallah, Fabio Casati, Gerti Kappel, and Gustavo Rossi (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg. 531–534. isbn:978-3-642-13911-6
[67]
Antonio Pecchia, Stefano Russo, and Santonu Sarkar. 2020. Assessing Invariant Mining Techniques for Cloud-Based Utility Computing Systems. IEEE Transactions on Services Computing, 13, 1 (2020), 44–58. https://doi.org/10.1109/TSC.2017.2679715
[68]
Leonard Richardson, Mike Amundsen, and Sam Ruby. 2013. RESTful Web APIs. O’Reilly Media, Inc. isbn:1449358063, 9781449358068
[69]
Sergio Segura, Juan C. Alonso, Alberto Martin-Lopez, Amador Durán, Javier Troya, and Antonio Ruiz-Cortés. 2022. Automated Generation of Metamorphic Relations for Query-Based Systems. In 2022 IEEE/ACM 7th International Workshop on Metamorphic Testing (MET). 48–55. https://doi.org/10.1145/3524846.3527338
[70]
Sergio Segura, Gordon Fraser, Ana B. Sanchez, and Antonio Ruiz-Cortés. 2016. A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering, 42, 9 (2016), 805–824. https://doi.org/10.1109/TSE.2016.2532875
[71]
Sergio Segura, José A Parejo, Javier Troya, and Antonio Ruiz-Cortés. 2018. Metamorphic Testing of RESTful Web APIs. IEEE Transactions on Software Engineering, 44, 11 (2018), 1083–1099. https://doi.org/10.1109/TSE.2017.2764464
[72]
Dimitri Stallenberg, Mitchell Olsthoorn, and Annibale Panichella. 2022. Improving Test Case Generation for REST APIs through Hierarchical Clustering. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE ’21). IEEE Press, 117–128. isbn:9781665403375 https://doi.org/10.1109/ASE51524.2021.9678586
[73]
Valerio Terragni, Gunel Jahangirova, Paolo Tonella, and Mauro Pezzè. 2020. Evolutionary Improvement of Assertion Oracles. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA. 1178–1189. isbn:9781450370431 https://doi.org/10.1145/3368089.3409758
[74]
Theofanis Vassiliou-Gioles. 2020. A simple, lightweight framework for testing RESTful services with TTCN-3. In 2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C). 498–505. https://doi.org/10.1109/QRS-C51114.2020.00089
[75]
Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RestTestGen: Automated Black-Box Testing of RESTful APIs. In International Conference on Software Testing, Verification and Validation.
[76]
Henry Vu, Tobias Fertig, and Peter Braun. 2018. Verification of Hypermedia Characteristic of RESTful Finite-State Machines. In Companion Proceedings of the The Web Conference 2018 (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE. 1881–1886. isbn:9781450356404 https://doi.org/10.1145/3184558.3191656
[77]
Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On Learning Meaningful Assert Statements for Unit Test Cases. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE ’20). Association for Computing Machinery, New York, NY, USA. 1398–1409. isbn:9781450371216 https://doi.org/10.1145/3377811.3380429
[78]
Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial Testing of RESTful APIs. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 426–437. isbn:9781450392211 https://doi.org/10.1145/3510003.3510151
[79]
S. Yoo and M. Harman. 2012. Regression Testing Minimization, Selection and Prioritization: A Survey. Softw. Test. Verif. Reliab., 22, 2 (2012), mar, 67–120. issn:0960-0833 https://doi.org/10.1002/stv.430
[80]
Hao Yu, Yiling Lou, Ke Sun, Dezhi Ran, Tao Xie, Dan Hao, Ying Li, Ge Li, and Qianxiang Wang. 2022. Automated Assertion Generation via Information Retrieval and Its Integration with Deep Learning. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22). Association for Computing Machinery, New York, NY, USA. 163–174. isbn:9781450392211 https://doi.org/10.1145/3510003.3510149
[81]
Juan Zhai, Yu Shi, Minxue Pan, Guian Zhou, Yongxiang Liu, Chunrong Fang, Shiqing Ma, Lin Tan, and Xiangyu Zhang. 2020. C2S: Translating Natural Language Comments to Formal Program Specifications. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA. 25–37. isbn:9781450370431 https://doi.org/10.1145/3368089.3409716
[82]
Man Zhang and Andrea Arcuri. 2021. Adaptive Hypermutation for Search-Based System Test Generation: A Study on REST APIs with EvoMaster. ACM Trans. Softw. Eng. Methodol., 31, 1 (2021), Article 2, sep, 52 pages. issn:1049-331X https://doi.org/10.1145/3464940
[83]
M. Zhang, A. Belhadi, and A. Arcuri. 2022. JavaScript Instrumentation for Search-Based Software Testing: A Study with RESTful APIs. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE Computer Society, Los Alamitos, CA, USA. 105–115. issn:2159-4848 https://doi.org/10.1109/ICST53961.2022.00022
[84]
Yuntong Zhang, Xiang Gao, Gregory J. Duck, and Abhik Roychoudhury. 2022. Program Vulnerability Repair via Inductive Inference. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2022). Association for Computing Machinery, New York, NY, USA. 691–702. isbn:9781450393799 https://doi.org/10.1145/3533767.3534387

Cited By

View all
  • (2024)Beyond REST: Introducing APIF for Comprehensive API Vulnerability FuzzingProceedings of the 27th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3678890.3678928(435-449)Online publication date: 30-Sep-2024
  • (2024)Brewing Up Reliability: Espresso Test Generation for Android Apps2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00025(185-196)Online publication date: 27-May-2024
  • (undefined)Test Oracle Automation in the Era of LLMsACM Transactions on Software Engineering and Methodology10.1145/3715107

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis
July 2023
1554 pages
ISBN:9798400702211
DOI:10.1145/3597926
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2023

Permissions

Request permissions for this article.

Check for updates

Badges

Author Tags

  1. REST APIs
  2. automated testing
  3. invariant detection
  4. test oracle

Qualifiers

  • Research-article

Funding Sources

  • MCIN/AEI /10.13039/501100011033/FEDER, UE
  • MCIN/AEI /10.13039/501100011033/FEDER, UE.
  • MCIN/AEI/10.13039/501100011033 and by European Union ?NextGenerationEU?/PRTR»

Conference

ISSTA '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)258
  • Downloads (Last 6 weeks)11
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Beyond REST: Introducing APIF for Comprehensive API Vulnerability FuzzingProceedings of the 27th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3678890.3678928(435-449)Online publication date: 30-Sep-2024
  • (2024)Brewing Up Reliability: Espresso Test Generation for Android Apps2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00025(185-196)Online publication date: 27-May-2024
  • (undefined)Test Oracle Automation in the Era of LLMsACM Transactions on Software Engineering and Methodology10.1145/3715107

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media