skip to main content
10.1145/3383845.3383890acmotherconferencesArticle/Chapter ViewAbstractPublication PagesiccmbConference Proceedingsconference-collections
research-article

Integrating Traversal and Visual-based Techniques to Initiate UI Exploration Scripting

Published: 05 April 2020 Publication History

Abstract

Initiating UI Test Automation project usually requires the testers who have related business knowledge, testing skill, and technical skill. Helping the testers start the project quickly and able to focus on other tasks in charge, this paper presents an approach to generate the initial UI test scripts by combining three main concepts of GUI automation including Scripting, Visual, and Traversal. The designed framework uses the data from the Accessibility API to extract target graphical elements on application under test (AUT). The traversal technique is applied for generating the sequences of interactions on all interactable objects based on Python GUI automated scripting (SikuliX) format. In case the API is not available on some OS/Applications, the computer vision technology will then be deployed on the Open Source Computer Vision Library (OpenCV) to extract the GUI elements. The implementation of the proposed framework is demonstrated via a couple of open-source applications.

References

[1]
Al´egroth, E., Feldt, R., and Kolstr¨om, P. 2016. Maintenance of automated test suites in industry: An empirical study on Visual GUI Testing. Information and Software Technology. 73, 66--80.
[2]
Rafi, D. M., Reddy, K., Moses, K., and Petersen, K. 2012. Benefits and Limitations of Automated Software Testing: Systematic Literature Review and Practitioner Survey. Automation of Software Test (AST), 7th International Workshop.
[3]
Al´egroth, E., and Feldt, R. 2017. On the long-term use of visual GUI testing in industrial practice: a case study. Empirical Software Engineering. 22, 2937--2971.
[4]
Grechanik, M., Xie, Q., and Fu, C. 2009. Maintaining and evolving GUI-directed test scripts. International Conference on Software Engineering. 408--418.
[5]
Al´egroth, E., and Feldt, R. 2014. Industrial application of visual GUI testing: Lessons learned. Continuous software engineering, 127--140.
[6]
Al´egroth, E., Gao, Z., Oliveira, R., and Memon A. 2015. Conceptualization and evaluation of component-based testing unified with visual GUI testing: An empirical study. 2015 IEEE 8th International Conference on Software Testing, Verification and Validation ICST, 1--10.
[7]
Dobslaw, F., Feldt, R., Michaelsson, D., Haar, P., de Oliveira Neto, F. G., Torkar, R. 2019. Estimating Return on Investment for GUI Test Automation Tools. arXiv:1907.03475 [cs.SE].
[8]
Tsung-Hsiang Chang, Tom Yeh, and Robert Miller, C. 2010. GUI Testing Using Computer Vision. CHI 2010: Pixel and Perception, (Atlanta, GA, USA, April 10--15, 2010), 1535--1544.
[9]
Vos, T. E.J., Molina, U. R., and Prasetya, W. 2016. Automated Testing at the User Interface level. ICTopen2016, Dutch ICT-research.
[10]
A. M. Memon. 2002. Gui testing: Pitfalls and process. Computer. 35, 87--88.
[11]
B. N. Nguyen, B. Robbins, I. Banerjee, and A. M. Memon, 2014. Guitar:an innovative tool for automated testing of gui-driven software. Automated Software Engineering. 21, 65--105.
[12]
Capgemini, Micro Focus, Sogeti. 2019. World Quality Report 2018--2019 TENTH EDITION. https://www.sogeti.com/globalassets/global/wqr-201819/wqr-2018-19_secured.pdf, Accessed: 24 Sep 2019
[13]
Nistér, D., and Stewénius, H. 2008. Linear time maximally stable extremal regions. Computer Vision-ECCV 2008, 183--196.

Index Terms

  1. Integrating Traversal and Visual-based Techniques to Initiate UI Exploration Scripting

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ICCMB '20: Proceedings of the 2020 the 3rd International Conference on Computers in Management and Business
    January 2020
    303 pages
    ISBN:9781450376778
    DOI:10.1145/3383845
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    In-Cooperation

    • Univ. of Manchester: University of Manchester
    • The Hong Kong Polytechnic: The Hong Kong Polytechnic University

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 April 2020

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Test automation
    2. UI scripting
    3. accessibility API
    4. computer vision
    5. graph traversal

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    ICCMB 2020

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 96
      Total Downloads
    • Downloads (Last 12 months)7
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 16 Jan 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media