research-article

Cytestion: Automated GUI Testing for Web Applications

Authors:

Thiago Santos de Moura,

Everton L. G. Alves,

Hugo Feitosa de Figueirêdo,

Cláudio de Souza BaptistaAuthors Info & Claims

SBES '23: Proceedings of the XXXVII Brazilian Symposium on Software Engineering

Pages 388 - 397

https://doi.org/10.1145/3613372.3613408

Published: 25 September 2023 Publication History

Abstract

Automated testing in web applications is becoming increasingly popular due to its cost-effectiveness, efficiency, and repeatability. Ensuring the quality of such software is vital, especially in scenarios where business requirements are uncertain and volatile. In this paper, we propose Cytestion, a tool that automatically explores a web application’s Graphical User Interface (GUI) to detect faults that cause visible failures, such as crashes, errors, and unexpected behaviors. For that, it applies a scriptless and progressive approach. To work as an oracle, Cytestion evaluates the output of each test case execution by considering the HTTP status of server requests, default and customized failure GUI messages, and console failure messages. We evaluated the effectiveness and cost of our approach in two empirical studies. The first study works with injected faults in open-source projects, while the second is a case study on eight industrial projects. Cytestion detected several faults in both studies and outperformed a state-of-the-art GUI testing tool.

References

[1]

Andrea Adamoli, Dmitrijs Zaparanuks, Milan Jovic, and Matthias Hauswirth. 2011. Automated GUI performance testing. Software Quality Journal 19, 4 (2011), 801–839.

Digital Library

[2]

Emil Alégroth, Robert Feldt, and Helena H Olsson. 2013. Transitioning manual system test suites to automated testing: An industrial case study. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. IEEE, IEEE, Luxembourg, Luxembourg, 56–65. https://doi.org/10.1109/ICST.2013.14

Digital Library

[3]

Emil Alégroth, Zebao Gao, Rafael Oliveira, and Atif Memon. 2015. Conceptualization and evaluation of component-based testing unified with visual gui testing: an empirical study. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). IEEE, IEEE, Graz, Austria, 1–10. https://doi.org/10.1109/ICST.2015.7102584

[4]

Francisco Almenar, Anna I Esparcia-Alcázar, Mirella Martínez, and Urko Rueda. 2016. Automated Testing of Web Applications with TESTAR: Lessons Learned Testing the Odoo Tool. In Search Based Software Engineering: 8th International Symposium, SSBSE 2016, Raleigh, NC, USA, October 8-10, 2016, Proceedings 8. Springer, Springer, Raleigh, North Carolina, USA, 218–223. https://doi.org/10.1007/978-3-319-47106-8_15

[5]

James H Andrews, Lionel C Briand, and Yvan Labiche. 2005. Is mutation an appropriate tool for testing experiments?. In Proceedings of the 27th international conference on Software engineering. Association for Computing Machinery, St. Louis MO USA, 402–411.

[6]

Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. In Proceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications. Association for Computing Machinery, New York, NY, USA, 641–660. https://doi.org/10.1145/2509136.2509549

Digital Library

[7]

Ishan Banerjee, Bao Nguyen, Vahid Garousi, and Atif Memon. 2013. Graphical user interface (GUI) testing: Systematic mapping and repository. Information and Software Technology 55, 10 (2013), 1679–1694.

Digital Library

[8]

Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2014. The oracle problem in software testing: A survey. IEEE transactions on software engineering 41, 5 (2014), 507–525.

Digital Library

[9]

Sebastian Bauersfeld, Tanja EJ Vos, Nelly Condori-Fernández, Alessandra Bagnato, and Etienne Brosse. 2014. Evaluating the TESTAR tool in an industrial case study. In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. Association for Computing Machinery, New York, NY, USA, 1–9.

Digital Library

[10]

Axel Bons, Beatriz Marín, Pekka Aho, and Tanja EJ Vos. 2023. Scripted and scriptless GUI testing for web applications: An industrial case. Information and Software Technology 158 (2023), 107172.

Digital Library

[11]

Emil Borjesson and Robert Feldt. 2012. Automated system testing using visual gui testing tools: A comparative study in industry. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. IEEE, 350–359.

Digital Library

[12]

Hatim Chahim, Mehmet Duran, Tanja EJ Vos, Pekka Aho, and Nelly Condori Fernandez. 2020. Scriptless testing at the GUI level in an industrial setting. In Research Challenges in Information Science: 14th International Conference, RCIS 2020, Limassol, Cyprus, September 23–25, 2020, Proceedings 14. Springer, 267–284.

[13]

Peter Chapman and David Evans. 2011. Automated black-box detection of side-channel vulnerabilities in web applications. In Proceedings of the 18th ACM conference on Computer and communications security. 263–274.

Digital Library

[14]

Kinga Dobolyi, Elizabeth Soechting, and Westley Weimer. 2011. Automating regression testing using web-based application similarities. International journal on software tools for technology transfer 13 (2011), 111–129. https://doi.org/10.1007/s10009-010-0170-x

[15]

Zebao Gao, Chunrong Fang, and Atif M Memon. 2015. Pushing the limits on automation in GUI regression testing. In 2015 IEEE 26th international symposium on software reliability engineering (ISSRE). IEEE, 565–575.

Digital Library

[16]

Boni García, Micael Gallego, Francisco Gortázar, and Mario Munoz-Organero. 2020. A survey of the selenium ecosystem. Electronics 9, 7 (2020), 1067.

[17]

Mark Grechanik, Qing Xie, and Chen Fu. 2009. Creating GUI testing tools using accessibility technologies. In 2009 International Conference on Software Testing, Verification, and Validation Workshops. IEEE, 243–250.

Digital Library

[18]

Mark Grechanik, Qing Xie, and Chen Fu. 2009. Maintaining and evolving GUI-directed test scripts. In 2009 IEEE 31st International Conference on Software Engineering. IEEE, 408–418.

Digital Library

[19]

Birgit Hofer, Bernhard Peischl, and Franz Wotawa. 2009. Gui savvy end-to-end testing with smart monkeys. In 2009 ICSE Workshop on Automation of Software Test. IEEE, 130–137.

[20]

Gunel Jahangirova. 2017. Oracle problem in software testing. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. 444–447.

Digital Library

[21]

René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 654–665.

Digital Library

[22]

Rebecca Krosnick and Steve Oney. 2021. Understanding the challenges and needs of programmers writing web automation scripts. In 2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 1–9.

[23]

Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2013. Capture-replay vs. programmable web testing: An empirical assessment during test case evolution. In 2013 20th Working Conference on Reverse Engineering (WCRE). IEEE, 272–281.

[24]

Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2014. Visual vs. DOM-based web locators: An empirical study. In Web Engineering: 14th International Conference, ICWE 2014, Toulouse, France, July 1-4, 2014. Proceedings 14. Springer, 322–340.

[25]

Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2016. Approaches and tools for automated end-to-end web testing. In Advances in Computers. Vol. 101. Elsevier, 193–237.

[26]

Grischa Liebel, Emil Alégroth, and Robert Feldt. 2013. State-of-practice in GUI-based system and acceptance testing: An industrial multiple-case study. In 2013 39th Euromicro Conference on Software Engineering and Advanced Applications. IEEE, 17–24.

Digital Library

[27]

Mirella Martínez, Anna I Esparcia-Alcázar, Tanja EJ Vos, Pekka Aho, and Joan Fons i Cors. 2018. Towards automated testing of the internet of things: Results obtained with the testar tool. In Leveraging Applications of Formal Methods, Verification and Validation. Distributed Systems: 8th International Symposium, ISoLA 2018, Limassol, Cyprus, November 5-9, 2018, Proceedings, Part III 8. Springer, 375–385.

[28]

Atif Memon, Ishan Banerjee, and Adithya Nagarajan. 2003. GUI ripping: Reverse engineering of graphical user interfaces for testing. In 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings. IEEE, 260–269.

[29]

Ali Mesbah, Arie van Deursen, and Stefan Lenselink. 2012. Crawling Ajax-Based Web Applications through Dynamic Analysis of User Interface State Changes. ACM Trans. Web 6, 1, Article 3 (mar 2012), 30 pages. https://doi.org/10.1145/2109205.2109208

Digital Library

[30]

Fatini Mobaraya, Shahid Ali, 2019. Technical Analysis of Selenium and Cypress as Functional Automation Framework for Modern Web Application Testing. In 9th International Conference on Computer Science.

[31]

Ad Mulders, Olivia Rodriguez Valdes, Fernando Pastor Ricós, Pekka Aho, Beatriz Marín, and Tanja EJ Vos. 2022. State model inference through the GUI using run-time test generation. In International Conference on Research Challenges in Information Science. Springer, 546–563.

[32]

Michel Nass, Emil Alégroth, and Robert Feldt. 2021. Why many challenges with GUI test automation (will) remain. Information and Software Technology 138 (2021), 106625.

[33]

Bao N. Nguyen, Bryan Robbins, Ishan Banerjee, and Atif Memon. 2014. GUITAR: An Innovative Tool for Automated Testing of GUI-Driven Software. Automated Software Engg. 21, 1 (mar 2014), 65–105. https://doi.org/10.1007/s10515-013-0128-9

Digital Library

[34]

Dario Olianas, Maurizio Leotta, and Filippo Ricca. 2022. SleepReplacer: a novel tool-based approach for replacing thread sleeps in selenium WebDriver test code. Software Quality Journal (2022), 1–33.

[35]

Rafael AP Oliveira, Upulee Kanewala, and Paulo A Nardi. 2014. Automated test oracles: State of the art, taxonomies, and trends. Advances in computers 95 (2014), 113–199.

[36]

Filippo Ricca, Maurizio Leotta, and Andrea Stocco. 2019. Three open problems in the context of e2e web testing and a vision: Neonate. In Advances in Computers. Vol. 113. Elsevier, 89–133.

[37]

Fernando Pastor Ricós, Arend Slomp, Beatriz Marín, Pekka Aho, and Tanja EJ Vos. 2023. Distributed state model inference for scriptless GUI testing. Journal of Systems and Software 200 (2023), 111645.

Digital Library

[38]

Olivia Rodríguez-Valdés, Tanja EJ Vos, Pekka Aho, and Beatriz Marín. 2021. 30 years of automated GUI testing: A bibliometric analysis. In Quality of Information and Communications Technology: 14th International Conference, QUATIC 2021, Algarve, Portugal, September 8–11, 2021, Proceedings 14. Springer, 473–488.

[39]

Carlos Gabriel Gomes de Melo Silva. 2019. Estudo comparativo de ferramentas de testes de ponta a ponta automatizados em sistemas web. B.S. thesis. Universidade Federal do Rio Grande do Norte.

[40]

Tommi Takala, Mika Katara, and Julian Harty. 2011. Experiences of system-level model-based GUI testing of an Android application. In 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation. IEEE, 377–386.

Digital Library

[41]

Paolo Tonella, Filippo Ricca, and Alessandro Marchetto. 2014. Recent advances in web testing. Advances in Computers 93 (2014), 1–51.

[42]

Tanja EJ Vos, Pekka Aho, Fernando Pastor Ricos, Olivia Rodriguez-Valdes, and Ad Mulders. 2021. TESTAR–scriptless testing through graphical user interface. Software Testing, Verification and Reliability 31, 3 (2021), e1771.

[43]

Tanja EJ Vos, Peter M Kruse, Nelly Condori-Fernández, Sebastian Bauersfeld, and Joachim Wegener. 2015. Testar: Tool support for test automation at the user interface level. International Journal of Information System Modeling and Design (IJISMD) 6, 3 (2015), 46–83.

Digital Library

[44]

Dacong Yan, Shengqian Yang, and Atanas Rountev. 2013. Systematic testing for resource leaks in Android applications. In 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 411–420. https://doi.org/10.1109/ISSRE.2013.6698894

[45]

Xun Yuan and Atif M. Memon. 2010. Generating Event Sequence-Based Test Cases Using GUI Runtime State Feedback. IEEE Transactions on Software Engineering 36, 1 (2010), 81–95. https://doi.org/10.1109/TSE.2009.68

Digital Library

Cited By

de Moura Tde Lima Mendes FAlves EDa Silva Neto IBaptista C(2024)An Automatic Approach for Uniquely Discovering Actionable Elements for Systematic GUI Testing in Web Applications2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS62785.2024.00012(13-24)Online publication date: 1-Jul-2024
https://doi.org/10.1109/QRS62785.2024.00012
Verma IWason RArora P(2024)TEST-Web: An Automated Web Application Testing Tool for Enhanced Efficiency and Consistency2024 3rd Edition of IEEE Delhi Section Flagship Conference (DELCON)10.1109/DELCON64804.2024.10866144(1-4)Online publication date: 21-Nov-2024
https://doi.org/10.1109/DELCON64804.2024.10866144

Index Terms

Cytestion: Automated GUI Testing for Web Applications
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation

Recommendations

Coverage criteria for GUI testing

A widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...
Towards a pattern language for model-based GUI testing
EuroPLoP '14: Proceedings of the 19th European Conference on Pattern Languages of Programs

Graphical user interfaces (GUIs) have become popular as they appear in everydays' software. GUIs have become an ideal way of interacting with computer programs, making the software friendlier to its users. GUIs have grown, and so has the usage of UI ...
Coverage criteria for GUI testing
ESEC/FSE-9: Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering

A widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SBES '23: Proceedings of the XXXVII Brazilian Symposium on Software Engineering

September 2023

570 pages

ISBN:9798400707872

DOI:10.1145/3613372

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 September 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SBES 2023

SBES 2023: XXXVII Brazilian Symposium on Software Engineering

September 25 - 29, 2023

Campo Grande, Brazil

Acceptance Rates

Overall Acceptance Rate 147 of 427 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
121
Total Downloads

Downloads (Last 12 months)65
Downloads (Last 6 weeks)9

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

de Moura Tde Lima Mendes FAlves EDa Silva Neto IBaptista C(2024)An Automatic Approach for Uniquely Discovering Actionable Elements for Systematic GUI Testing in Web Applications2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS62785.2024.00012(13-24)Online publication date: 1-Jul-2024
https://doi.org/10.1109/QRS62785.2024.00012
Verma IWason RArora P(2024)TEST-Web: An Automated Web Application Testing Tool for Enhanced Efficiency and Consistency2024 3rd Edition of IEEE Delhi Section Flagship Conference (DELCON)10.1109/DELCON64804.2024.10866144(1-4)Online publication date: 21-Nov-2024
https://doi.org/10.1109/DELCON64804.2024.10866144

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten