skip to main content
10.1145/3611643.3616324acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Semantic Test Repair for Web Applications

Published: 30 November 2023 Publication History

Abstract

Automation testing is widely used in the functional testing of web applications. However, during the evolution of web applications, such web test scripts tend to break. It is essential to repair such broken test scripts to make regression testing run successfully. As manual repairing is time-consuming and expensive, researchers focus on automatic repairing techniques. Empirical study shows that the web element locator is the leading cause of web test breakages. Most existing repair techniques utilize Document Object Model attributes or visual appearances of elements to find their location but neglect their semantic information.
This paper proposes a novel semantic repair technique called Semantic Test Repair (Semter) for web test repair. Our approach captures relevant semantic information from test executions on the application’s basic version and locates target elements by calculating semantic similarity between elements to repair tests. Our approach can also repair test workflow due to web page additions or deletions by a local exploration in the updated version. We evaluated the efficacy of our technique on six real-world web applications compared with three baselines. Experimental results show that Semter achieves an 84% average repair ratio within an acceptable time cost, significantly outperforming the state-of-the-art web test repair techniques.

Supplementary Material

Video (fse23main-p728-p-video.mp4)
"Why does my program fail? We present a novel and general technique to automatically determine failure causes and conditions, using logical properties over input elements: “The program fails if and only if int(⟨length⟩) > len(⟨payload⟩) holds - that is, the given ⟨length⟩ is larger than the ⟨payload⟩ length.” Our AVICENNA prototype uses modern techniques for inferring properties of passing and failing inputs and validating and refining hypotheses by having a constraint solver generate supporting test cases to obtain such diagnoses. As a result, AVICENNA produces crisp and expressive diagnoses even for complex failure conditions, considerably improving over the state of the art with diagnoses close to those of human experts."

References

[1]
[n. d.]. AddressBook. https://sourceforge.net/projects/php-addressbook/.
[2]
[n. d.]. Claroline. https://sourceforge.net/projects/claroline/.
[3]
[n. d.]. Collabtive. https://sourceforge.net/projects/collabtive/.
[4]
[n. d.]. MantisBT. https://sourceforge.net/projects/mantisbt/.
[5]
[n. d.]. Meeting Room Booking System. https://sourceforge.net/projects/mrbs/.
[6]
[n. d.]. Password-Manager. https://sourceforge.net/projects/ phppasswordmanager/.
[7]
Farnaz Behrang and Alessandro Orso. 2018. Test migration for eficient largescale assessment of mobile app coding assignments. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, Amsterdam, The Netherlands, July 16-21, 2018, Frank Tip and Eric Bodden (Eds.). ACM, 164-175. https://doi.org/10.1145/3213846.3213854
[8]
Farnaz Behrang and Alessandro Orso. 2019. Test Migration Between Mobile Apps with Similar Functionality. In 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019. IEEE, 54-65. https://doi.org/10.1109/ASE. 2019.00016
[9]
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
[10]
Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020. Unblind your apps: predicting naturallanguage labels for mobile GUI components by deep learning. In ICSE '20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June-19 July, 2020, Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 322-334. https://doi.org/10.1145/3377811.3380327
[11]
Shauvik Roy Choudhary, Dan Zhao, Husayn Versee, and Alessandro Orso. 2011. Water: Web application test repair. In Proceedings of the First International Workshop on End-to-End Test Script Engineering. 24-29.
[12]
B. Daniel, Q. Luo, M. Mirzaaghaei, D. Marinov, and M Pezzè. 2011. Automated GUI refactoring and test script repair. ACM ( 2011 ).
[13]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171-4186. https://doi.org/10.18653/v1/n19-1423
[14]
Zebao Gao, Zhenyu Chen, Yunxiao Zou, and Atif M. Memon. 2016. SITAR: GUI Test Script Repair. IEEE Trans. Software Eng. 42, 2 ( 2016 ), 170-186. https: //doi.org/10.1109/TSE. 2015.2454510
[15]
Mark Grechanik, Qing Xie, and Chen Fu. 2009. Maintaining and evolving GUIdirected test scripts. In 31st International Conference on Software Engineering, ICSE 2009, May 16-24, 2009, Vancouver, Canada, Proceedings. IEEE, 408-418. https: //doi.org/10.1109/ICSE. 2009.5070540
[16]
Mouna Hammoudi, Gregg Rothermel, and Andrea Stocco. 2016. WATERFALL: an incremental approach for repairing record-replay tests of web applications. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016, Thomas Zimmermann, Jane Cleland-Huang, and Zhendong Su (Eds.). ACM, 751-762. https://doi.org/10.1145/2950290.2950294
[17]
Mouna Hammoudi, Gregg Rothermel, and Paolo Tonella. 2016. Why do Record/Replay Tests of Web Applications Break?. In 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, Chicago, IL, USA, April 11-15, 2016. IEEE Computer Society, 180-190. https://doi.org/10.1109/ ICST. 2016.16
[18]
Mark Harman and Nadia Alshahwan. 2008. Automated Session Data Repair for Web Application Regression Testing. In First International Conference on Software Testing, Verification, and Validation, ICST 2008, Lillehammer, Norway, April 9-11, 2008. IEEE Computer Society, 298-307. https://doi.org/10.1109/ICST. 2008.56
[19]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 770-778. https://doi.org/10.1109/CVPR. 2016.90
[20]
Si Huang, Myra B. Cohen, and Atif M. Memon. 2010. Repairing GUI Test Suites Using a Genetic Algorithm. In Third International Conference on Software Testing, Verification and Validation, ICST 2010, Paris, France, April 7-9, 2010. IEEE Computer Society, 245-254. https://doi.org/10.1109/ICST. 2010.39
[21]
Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2014. Visual vs. DOM-Based Web Locators: An Empirical Study. In Web Engineering, 14th International Conference, ICWE 2014, Toulouse, France, July 1-4, 2014. Proceedings (Lecture Notes in Computer Science, Vol. 8541 ), Sven Casteleyn, Gustavo Rossi, and Marco Winckler (Eds.). Springer, 322-340. https://doi.org/10.1007/978-3-319-08245-5_19
[22]
Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2014. Reducing Web Test Cases Aging by Means of Robust XPath Locators. In 25th IEEE International Symposium on Software Reliability Engineering Workshops, ISSRE Workshops, Naples, Italy, November 3-6, 2014. IEEE Computer Society, 449-454. https://doi.org/10.1109/ISSREW. 2014.17
[23]
Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2015. Using Multi-Locators to Increase the Robustness of Web Test Cases. In 8th IEEE International Conference on Software Testing, Verification and Validation, ICST 2015, Graz, Austria, April 13-17, 2015. IEEE Computer Society, 1-10. https: //doi.org/10.1109/ICST. 2015.7102611
[24]
Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2016. Robula+: an algorithm for generating robust XPath locators for web testing. J. Softw. Evol. Process. 28, 3 ( 2016 ), 177-204. https://doi.org/10.1002/smr.1771
[25]
Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2018. Pesto: Automated migration of DOM-based Web tests towards the visual approach. Softw. Test. Verification Reliab. 28, 4 ( 2018 ). https://doi.org/10.1002/stvr.1665
[26]
Jun-Wei Lin, Reyhaneh Jabbarvand, and Sam Malek. 2019. Test Transfer Across Mobile Apps Through Semantic Mapping. In 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019. IEEE, 42-53. https://doi.org/10.1109/ASE. 2019.00015
[27]
Peng Liu, Xiangyu Zhang, Marco Pistoia, Yunhui Zheng, Manoel Marques, and Lingfei Zeng. 2017. Automatic text input generation for mobile testing. In Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017, Sebastián Uchitel, Alessandro Orso, and Martin P. Robillard (Eds.). IEEE / ACM, 643-653. https://doi.org/10.1109/ICSE. 2017.65
[28]
Zhenyue Long, Guoquan Wu, Xiaojiang Chen, Wei Chen, and Jun Wei. 2020. WebRR: self-replay enhanced robust record/replay for web application testing. In ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 1498-1508. https://doi.org/10.1145/3368089.3417069
[29]
Leonardo Mariani, Ali Mohebbi, Mauro Pezzè, and Valerio Terragni. 2021. Semantic matching of GUI events for test reuse: are we there yet?. In ISSTA '21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, Denmark, July 11-17, 2021, Cristian Cadar and Xiangyu Zhang (Eds.). ACM, 177-190. https://doi.org/10.1145/3460319.3464827
[30]
Leonardo Mariani, Mauro Pezzè, and Daniele Zuddas. 2018. Augusto: exploiting popular functionalities for the generation of semantic GUI tests with Oracles. In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27-June 03, 2018, Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (Eds.). ACM, 280-290. https://doi.org/10. 1145/3180155.3180162
[31]
Forough Mehralian, Navid Salehnamadi, and Sam Malek. 2021. Data-driven accessibility repair revisited: on the efectiveness of generating labels for icons in Android apps. In ESEC/FSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (Eds.). ACM, 107-118. https://doi.org/10.1145/3468264. 3468604
[32]
Atif M. Memon. 2008. Automatically repairing event sequence-based GUI test suites for regression testing. ACM Trans. Softw. Eng. Methodol. 18, 2 ( 2008 ), 4 : 1-4 : 36. https://doi.org/10.1145/1416563.1416564
[33]
Ali Mesbah. 2015. Advances in Testing JavaScript-Based Web Applications. Adv. Comput. 97 ( 2015 ), 201-235. https://doi.org/10.1016/bs.adcom. 2014. 12.003
[34]
Shabnam Mirshokraie, Ali Mesbah, and Karthik Pattabiraman. 2015. JSEFT: Automated Javascript Unit Test Generation. In 8th IEEE International Conference on Software Testing, Verification and Validation, ICST 2015, Graz, Austria, April 13-17, 2015. IEEE Computer Society, 1-10. https://doi.org/10.1109/ICST. 2015.7102595
[35]
Minxue Pan, Tongtong Xu, Yu Pei, Zhong Li, Tian Zhang, and Xuandong Li. 2022. GUI-Guided Test Script Repair for Mobile Apps. IEEE Trans. Software Eng. 48, 3 ( 2022 ), 910-929. https://doi.org/10.1109/TSE. 2020.3007664
[36]
Andreas Rau, Jenny Hotzkow, and Andreas Zeller. 2018. Transferring Tests Across Web Applications. In Web Engineering-18th International Conference, ICWE 2018, Cáceres, Spain, June 5-8, 2018, Proceedings (Lecture Notes in Computer Science, Vol. 10845 ), Tommi Mikkonen, Ralf Klamma, and Juan Hernández (Eds.). Springer, 50-64. https://doi.org/10.1007/978-3-319-91662-0_4
[37]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 3980-3990. https://doi.org/10.18653/v1/ D19-1410
[38]
Danny Roest, Ali Mesbah, and Arie van Deursen. 2010. Regression Testing Ajax Applications: Coping with Dynamism. In Third International Conference on Software Testing, Verification and Validation, ICST 2010, Paris, France, April 7-9, 2010. IEEE Computer Society, 127-136. https://doi.org/10.1109/ICST. 2010.59
[39]
Fei Shao, Rui Xu, Wasif Arman Haque, Jingwei Xu, Ying Zhang, Wei Yang, Yanfang Ye, and Xusheng Xiao. 2021. WebEvo: taming web application evolution via detecting semantic structure changes. In ISSTA '21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, Denmark, July 11-17, 2021, Cristian Cadar and Xiangyu Zhang (Eds.). ACM, 16-28. https://doi.org/10.1145/3460319.3464800
[40]
Andrea Stocco, Rahulkrishna Yandrapally, and Ali Mesbah. 2018. Visual web test repair. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018, Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (Eds.). ACM, 503-514. https://doi.org/10.1145/3236024.3236063
[41]
Saghar Talebipour, Yixue Zhao, Luka Dojcilovic, Chenggang Li, and Nenad Medvidovic. 2021. UI Test Migration Across Mobile Platforms. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021. IEEE, 756-767. https://doi.org/10.1109/ASE51524. 2021.9678643
[42]
Suresh Thummalapenta, K. Vasanta Lakshmi, Saurabh Sinha, Nishant Sinha, and Satish Chandra. 2013. Guided test generation for web applications. In 35th International Conference on Software Engineering, ICSE ' 13, San Francisco, CA, USA, May 18-26, 2013, David Notkin, Betty H. C. Cheng, and Klaus Pohl (Eds.). IEEE Computer Society, 162-171. https://doi.org/10.1109/ICSE. 2013.6606562
[43]
Susana M. Vieira, Uzay Kaymak, and João M. C. Sousa. 2010. Cohen's kappa coeficient as a performance measure for feature selection. In FUZZ-IEEE 2010, IEEE International Conference on Fuzzy Systems, Barcelona, Spain, 18-23 July, 2010, Proceedings. IEEE, 1-8. https://doi.org/10.1109/FUZZY. 2010.5584447
[44]
Tongtong Xu, Minxue Pan, Yu Pei, Guiyin Li, Xia Zeng, Tian Zhang, Yuetang Deng, and Xuandong Li. 2021. GUIDER: GUI structure and vision co-guided test script repair for Android apps. In ISSTA '21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, Denmark, July 11-17, 2021, Cristian Cadar and Xiangyu Zhang (Eds.). ACM, 191-203. https://doi.org/ 10.1145/3460319.3464830
[45]
Rahulkrishna Yandrapally, Suresh Thummalapenta, Saurabh Sinha, and Satish Chandra. 2014. Robust test automation using contextual clues. In International Symposium on Software Testing and Analysis, ISSTA '14, San Jose, CA, USA-July 21-26, 2014, Corina S. Pasareanu and Darko Marinov (Eds.). ACM, 304-314. https://doi.org/10.1145/2610384.2610390
[46]
Juyeon Yoon, Seungjoon Chung, Kihyuck Shin, Jinhan Kim, Shin Hong, and Shin Yoo. 2022. Repairing Fragile GUI Test Cases Using Word and Layout Embedding. In 15th IEEE Conference on Software Testing, Verification and Validation, ICST 2022, Valencia, Spain, April 4-14, 2022. IEEE, 291-301. https://doi.org/10.1109/ ICST53961. 2022.00038

Cited By

View all
  • (2024)Towards a Robust Waiting Strategy for Web GUI Testing for an Industrial Software SystemProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695269(2065-2076)Online publication date: 27-Oct-2024
  • (2024)Enhancing Web Test Script Repair Using Integrated UI Structural and Visual Information2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00018(75-86)Online publication date: 6-Oct-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
November 2023
2215 pages
ISBN:9798400703270
DOI:10.1145/3611643
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 November 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. GUI Testing
  2. Semantic Similarity
  3. Test Repair
  4. Web Testing

Qualifiers

  • Research-article

Funding Sources

  • National Science Foundation of China

Conference

ESEC/FSE '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)128
  • Downloads (Last 6 weeks)7
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Towards a Robust Waiting Strategy for Web GUI Testing for an Industrial Software SystemProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695269(2065-2076)Online publication date: 27-Oct-2024
  • (2024)Enhancing Web Test Script Repair Using Integrated UI Structural and Visual Information2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00018(75-86)Online publication date: 6-Oct-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media