skip to main content
10.1145/3474624.3474640acmotherconferencesArticle/Chapter ViewAbstractPublication PagessbesConference Proceedingsconference-collections
research-article

Experiences and Practices in GUI Functional Testing: A Software Practitioners’ View

Published: 05 October 2021 Publication History

Abstract

Software testing is an important activity to ensure software quality. A widely performed type of testing is GUI (Graphical User Interface) functional testing, where the aim is to evaluate the software functionality through their graphic interface. We can find in the literature techniques and solutions for handling GUI functional testing in the last years. However, studies that deeply analyze how professionals in the industry perform GUI functional testing are still lacking. In this study, we investigated how testing professionals perform GUI functional testing. We surveyed 222 professionals from different positions and roles who perform those tests. The results indicate that many professionals perform GUI functional testing without proper tool support. They also pointed out that GUI test automation tools have limitations. This study presents preliminary results to understand the Brazilian companies’ scenario on the difficulties of GUI functional testing.

References

[1]
Emil Alégroth, Robert Feldt, and Pirjo Kolström. 2016. Maintenance of automated test suites in industry: An empirical study on Visual GUI Testing. Inf. Softw. Technol. 73(2016), 66–80.
[2]
Emil Alégroth, Robert Feldt, and Helena Holmström Olsson. 2013. Transitioning Manual System Test Suites to Automated Testing: An Industrial Case Study. In Sixth IEEE International Conference on Software Testing, Verification and Validation(ICST’13). IEEE Computer Society, 56–65.
[3]
Emil Alégroth, Robert Feldt, and Lisa Ryrholm. 2015. Visual GUI testing in practice: challenges, problems and limitations. Empir. Softw. Eng. 20, 3 (2015), 694–744.
[4]
Emil Alegroth, Zebao Gao, Rafael Oliveira, and Atif Memon. 2015. Conceptualization and Evaluation of Component-Based Testing Unified with Visual GUI Testing: An Empirical Study. In IEEE 8th International Conference on Software Testing, Verification and Validation(ICST’15).
[5]
Amira Ali, Huda Amin Maghawry, and Nagwa Badr. 2018. Automated parallel GUI testing as a service for mobile applications. Journal of Software: Evolution and Process 30, 10 (2018).
[6]
AltexSoft. 2020. Quality-Assurance-Quality-Control-and-Testing-Whitepaper. https://behave.readthedocs.io/en/stable/philosophy.html
[7]
Domenico Amalfitano, Nicola Amatucci, Atif M. Memon, Porfirio Tramontana, and Anna Rita Fasolino. 2017. A general framework for comparing automatic testing techniques of Android mobile apps. J. Syst. Softw. 125(2017), 322–343.
[8]
Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M. Memon. 2012. Using GUI Ripping for Automated Testing of Android Applications. In 27th IEEE/ACM International Conference on Automated Software Engineering. ACM, 258–261.
[9]
Luca Ardito, Riccardo Coppola, Maurizio Morisio, and Marco Torchiano. 2019. Espresso vs. EyeAutomate: An Experiment for the Comparison of Two Generations of Android GUI Testing(EASE ’19). ACM, 13–22.
[10]
Yauhen Leanidavich Arnatovich and Lipo Wang. 2018. A Systematic Literature Review of Automated Techniques for Functional GUI Testing of Mobile Applications. arxiv:cs.SE/1812.11470
[11]
Ishan Banerjee, Bao Nguyen, Vahid Garousi, and Atif Memon. 2013. Graphical User Interface (GUI) Testing: Systematic Mapping and Repository. Information and Software Technology 55, 10 (Oct. 2013), 1679–1694.
[12]
Richard Jones Benno Rice and Jens Engel Revision. 2017. Behavior Driven Development. https://behave.readthedocs.io/en/stable/philosophy.html
[13]
Stefan Berner, Roland Weber, and Rudolf K. Keller. 2005. Observations and Lessons Learned from Automated Testing. In Proceedings of the 27th International Conference on Software Engineering(ICSE ’05). ACM, 571–579.
[14]
Jehad Al Dallal. 2009. Automation of object-oriented framework application testing. In 2009 5th IEEE GCC Conference Exhibition. IEEE, 1–5.
[15]
Omar El Ariss, Dianxiang Xu, Santosh Dandey, Brad Vender, Phil McClean, and Brian Slator. 2010. A Systematic Capture and Replay Strategy for Testing Complex GUI Based Java Applications. In Proceedings of the 2010 Seventh International Conference on Information Technology: New Generations(ITNG ’10). IEEE, USA.
[16]
Vahid Garousi, Wasif Afzal, Adem Çağlar, Berk Işık, Berker Baydan, Seçkin Çaylak, Ahmet Zeki Boyraz, Burak Yolaçan, and Kadir Herkiloğlu. 2017. Comparing Automated Visual GUI Testing Tools: An Industrial Case Study. In Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing(Paderborn, Germany) (A-TEST 2017). ACM, 21–28.
[17]
Vahid Garousi, Wasif Afzal, Adem Çağlar, İhsan Berk Işık, Berker Baydan, Seçkin Çaylak, Ahmet Zeki Boyraz, Burak Yolaçan, and Kadir Herkiloğlu. 2020. Visual GUI testing in practice: An extended industrial case study. arxiv:cs.SE/2005.09303
[18]
Mark Grechanik, Qing Xie, and Chen Fu. 2009. Creating GUI Testing Tools Using Accessibility Technologies. In International Conference on Software Testing, Verification, and Validation Workshops. IEEE, 243–250.
[19]
S Gunasekaran and V Bargavi. 2015. Survey on automation testing tools for mobile applications. International Journal of Advanced Engineering Research and Science 2, 11(2015), 2349–6495.
[20]
Marek Janicki, Mika Katara, and Tuula Pääkkönen. 2012. Obstacles and opportunities in deploying model-based GUI testing of mobile software: a survey. Software Testing, Verification and Reliability 22, 5(2012), 313–341.
[21]
Katja Karhu, Tiina Repo, Ossi Taipale, and Kari Smolander. 2009. Empirical Observations on Software Testing Automation. In International Conference on Software Testing Verification and Validation.
[22]
Onur Kilincceker, Alper Silistre, Fevzi Belli, and Moharram Challenger. 2021. Model-Based Ideal Testing of GUI Programs–Approach and Case Studies. IEEE Access 9(2021), 68966–68984.
[23]
Meile Posthuma Klaus Olsen and Stephanie Ulrich. 2019. International Software Testing Qualifications Board (third ed.). International Software Testing Qualifications Board. https://www.istqb.org/downloads/send/2-foundation-level-documents/281-istqb-ctfl-syllabus-2018-v3-1.html
[24]
Shin-Jie Lee, Yu-Xian Chen, Shang-Pin Ma, and Wen-Tin Lee. 2018. Test Command Auto-Wait Mechanisms for Record and Playback-Style Web Application Testing. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol. 2. 75–80.
[25]
Andreas Leitner, Ilinca Ciupa, Bertrand Meyer, and Mark Howard. 2007. Reconciling Manual and Automated Testing: The AutoTest Experience. In 40th Annual Hawaii International Conference on System Sciences(HICSS ’07).
[26]
Mario Linares-Vásquez, Cárlos Bernal-Cardenas, Kevin Moran, and Denys Poshyvanyk. 2017. How do Developers Test Android Applications?. In IEEE International Conference on Software Maintenance and Evolution(ICSME ’2017). 613–622.
[27]
A.M. Memon, M.E. Pollack, and M.L. Soffa. 2001. Hierarchical GUI test case generation using automated planning. IEEE Transactions on Software Engineering 27, 2 (2001), 144–155.
[28]
Duaa R. Mohammad, Sajedah Al-Momani, Yahya M. Tashtoush, and Mohammad Alsmirat. 2019. A Comparative Analysis of Quality Assurance Automated Testing Tools for Windows Mobile Applications. In 2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC).
[29]
Ioannis Parissis and Farid Ouabdesselam. 1996. Specification-based Testing of Synchronous Software. In SIGSOFT ’96, Proceedings of the Fourth ACM SIGSOFT Symposium on Foundations of Software Engineering. ACM, 127–134.
[30]
I. A. Qureshi and A. Nadeem. 2013. GUI Testing Techniques: A Survey. International Journal of Future Computer and Communication (2013), 142–146.
[31]
Dudekula Mohammad Rafi, Katam Reddy Kiran Moses, Kai Petersen, and Mika V. Mäntylä. 2012. Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. In 7th International Workshop on Automation of Software Test(AST ’2012). 36–42.
[32]
Rudolf Ramler and Klaus Wolfmaier. 2006. Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity Cost. In Proceedings of the 2006 International Workshop on Automation of Software Test (Shanghai, China) (AST ’06). ACM, 85–91.
[33]
F. Saglietti and F. Pinte. 2010. Automated Unit and Integration Testing for Component-Based Software Systems, In International Workshop on Security and Dependability for Resource Constrained Embedded Systems. ACM.
[34]
Mary Sánchez-Gordón, Laxmi Rijal, and Ricardo Colomo-Palacios. 2020. Beyond Technical Skills in Software Testing: Automated versus Manual Testing. In IEEE/ACM 42nd International Conference on Software Engineering Workshops (Seoul, Republic of Korea) (ICSEW’20). ACM, 161–164.
[35]
R. M. Sharma. 2014. Quantitative Analysis of Automation and Manual Testing. Technical Report Volume 4. International Journal of Engineering and Innovative Technology (IJEIT).
[36]
Pradeep Sharma Shruti Malve. 2017. Investigation of Manual and Automation Testing using Assorted Approaches. International Journal of Scientific Research in Computer Science and Engineering 5 (4 2017), 81–87. Issue 2. https://www.isroset.org/journal/IJSRCSE/full_paper_view.php?paper_id=408
[37]
ISO 25000 STANDARDS. 2011. ISO/IEC 25010. Retrieved May 03, 2021 from https://iso25000.com/index.php/en/iso-25000-standards/iso-25010?start=6
[38]
Porfirio Tramontana, Domenico Amalfitano, Nicola Amatucci, and Anna Rita Fasolino. 2019. Automated functional testing of mobile applications: a systematic mapping study. Softw. Qual. J. 27, 1 (2019), 149–201.

Cited By

View all
  • (2023)Applicability of the ViMoTest Approach for Automated GUI Testing: A Field Study2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C)10.1109/MODELS-C59198.2023.00131(821-830)Online publication date: 1-Oct-2023
  • (2022)We Don’t Need No Real Users?! Surveying the Adoption of User-less Automation Tools by UI Design PractitionersWeb Engineering10.1007/978-3-031-09917-5_28(406-414)Online publication date: 5-Jul-2022
  1. Experiences and Practices in GUI Functional Testing: A Software Practitioners’ View

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    SBES '21: Proceedings of the XXXV Brazilian Symposium on Software Engineering
    September 2021
    473 pages
    ISBN:9781450390613
    DOI:10.1145/3474624
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 October 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. GUI funcional testing
    2. Software testing
    3. software testers
    4. surveys

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    SBES '21
    SBES '21: Brazilian Symposium on Software Engineering
    September 27 - October 1, 2021
    Joinville, Brazil

    Acceptance Rates

    Overall Acceptance Rate 147 of 427 submissions, 34%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)29
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Applicability of the ViMoTest Approach for Automated GUI Testing: A Field Study2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C)10.1109/MODELS-C59198.2023.00131(821-830)Online publication date: 1-Oct-2023
    • (2022)We Don’t Need No Real Users?! Surveying the Adoption of User-less Automation Tools by UI Design PractitionersWeb Engineering10.1007/978-3-031-09917-5_28(406-414)Online publication date: 5-Jul-2022

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media