skip to main content
10.1145/3377811.3380431acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

RoScript: a visual script driven truly non-intrusive robotic testing system for touch screen applications

Published: 01 October 2020 Publication History

Abstract

Existing intrusive test automation techniques for touch screen applications (e.g., Appium and Sikuli) are difficult to work on many closed or uncommon systems, such as a GoPro. Being non-intrusive can largely extend the application scope of the test automation techniques. To this end, this paper presents RoScript, a truly non-intrusive test-script-driven robotic testing system for test automation of touch screen applications. RoScript leverages visual test scripts to express GUI actions on a touch screen application and uses a physical robot to drive automated test execution. To reduce the test script creation cost, a non-intrusive computer vision based technique is also introduced in RoScript to automatically record touch screen actions into test scripts from videos of human actions on the device under test. RoScript is applicable to touch screen applications running on almost arbitrary platforms, whatever the underlying operating systems or GUI frameworks are. We conducted experiments applying it to automate the testing of 21 touch screen applications on 6 different devices. The results show that RoScript is highly usable. In the experiments, it successfully automated 104 test scenarios containing over 650 different GUI actions on the subject applications. RoScript accurately performed GUI actions on over 90% of the test script executions and accurately recorded about 85% of human screen click actions into test code.

References

[1]
2020. Appium: Automation for iOS and Android apps. http://appium.io
[2]
2020. EBB Command Set. http://evil-mad.github.io/EggBot/ebb.html
[3]
2020. EggPlant. https://www.eggplantsoftware.com/
[4]
2020. OpenCV. https://opencv.org/
[5]
2020. OptoFidelity test solutions. https://www.optofidelity.com/test-solutions/
[6]
2020. Robotium. https://github.com/RobotiumTech/robotium
[7]
2020. RoScript evaluation artifacts. https://github.com/juqian/ICSE2020-RoScript
[8]
2020. UIAutomator. http://developer.android.com/tools/testingsupport-library
[9]
2020. VNC. https://en.wikipedia.org/wiki/Virtual_Network_Computing
[10]
Emil Alégroth and Robert Feldt. 2017. On the long-term use of visual GUI testing in industrial practice: a case study. Empirical Software Engineering 22, 6 (2017), 2937--2971.
[11]
Emil Alégroth, Robert Feldt, and Lisa Ryrholm. 2015. Visual GUI testing in practice: challenges, problems and limitations. Empirical Software Engineering 20, 3 (2015), 694--744.
[12]
Emil Alégroth, Michel Nass, and Helena H. Olsson. 2013. JAutomate: A tool for system and acceptance test automation. In IEEE 6th International Conference on Software Testing, Verification and Validation (ICST). 439--446.
[13]
Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Bryan Dzung Ta, and Atif M Memon. 2014. MobiGUITAR: Automated model-based testing of mobile apps. IEEE software 32, 5 (2014), 53--59.
[14]
Andreas Bihlmaier and Heinz Wörn. 2014. Robot unit testing. In International Conference on Simulation, Modeling, and Programming for Autonomous Robots. Springer, 255--266.
[15]
Emil Borjesson and Robert Feldt. 2012. Automated system testing using visual GUI testing tools: A comparative study in industry. In IEEE 5th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 350--359.
[16]
Tsung-Hsiang Chang, Tom Yeh, and Robert C Miller. 2010. GUI testing using computer vision. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1535--1544.
[17]
Mathieu Collet, Arnaud Gotlieb, Nadjib Lazaar, and Morten Mossige. 2019. Stress testing of single-arm robots through constraint-based generation of continuous trajectories. In IEEE International Conference On Artificial Intelligence Testing. IEEE, 121--128.
[18]
Karthikeyan B. Dhanapal, K Sai Deepak, Saurabh Sharma, Sagar P. Joglekar, Aditya Narang, Aditya Vashistha, Paras Salunkhe, Harikrishna G.N. Rai, Arun A. Somasundara, and Sanjoy Paul. 2012. An innovative system for remote and automated testing of mobile phone applications. In Annual SRII Global Conference. IEEE, 44--54.
[19]
Mattia Fazzini, Eduardo Noronha de A Freitas, Shauvik Roy Choudhary, and Alessandro Orso. 2017. Barista: A technique for recording, encoding, and running platform independent Android tests. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 149--160.
[20]
Lorenzo Gomez, Iulian Neamtiu, Tanzirul Azim, and Todd Millstein. 2013. RERAN: Timing-and touch-sensitive record and replay for Android. In Proceedings of the International Conference on Software Engineering (ICSE). IEEE, 72--81.
[21]
Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In the 41st International Conference on Software Engineering (ICSE). IEEE, 269--280.
[22]
Jiaqi Guo, Shuyue Li, Jian-Guang Lou, Zijiang Yang, and Ting Liu. 2019. SARA: self-replay augmented record and replay for Android in industrial cases. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, 90--100.
[23]
Matthew Halpern, Yuhao Zhu, Ramesh Peri, and Vijay Janapa Reddi. 2015. Mosaic: cross-platform user-interaction record and replay for the fragmented Android ecosystem. In IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 215--224.
[24]
Yongjian Hu, Tanzirul Azim, and Iulian Neamtiu. 2015. Versatile yet lightweight record-and-replay for Android. In Proceedings of the Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA). ACM, 349--366.
[25]
Hengle Jiang, Sebastian Elbaum, and Carrick Detweiler. 2017. Inferring and monitoring invariants in robotic systems. Autonomous Robots 41, 4 (2017), 1027--1046.
[26]
Teemu Kanstrén, Pekka Aho, Arttu Lämsä, Henar Martin, Jussi Liikka, and Miska Seppänen. 2015. Robot-assisted smartphone performance testing. In IEEE International Conference on Technologies for Practical Robot Applications. IEEE, 1--6.
[27]
Pingfan Kong, Li Li, Jun Gao, Kui Liu, Tegawendé F Bissyandé, and Jacques Klein. 2018. Automated testing of Android apps: A systematic literature review. IEEE Transactions on Reliability 68, 1 (2018), 45--66.
[28]
Wing Lam, Zhengkai Wu, Dengfeng Li, Wenyu Wang, Haibing Zheng, Hui Luo, Peng Yan, Yuetang Deng, and Tao Xie. 2017. Record and replay for Android: are we there yet in industrial cases?. In Proceedings of the Joint Meeting on Foundations of Software Engineering (FSE). ACM, 854--859.
[29]
Mario Linares-Vásquez, Kevin Moran, and Denys Poshyvanyk. 2017. Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing. In IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 399--410.
[30]
Jin-Feng Luan, Dror Saaroni, and Xiao-Ming Hu. 2018. Generating software test script from video. US Patent 10,019,346.
[31]
Niloofar Mansoor, Jonathan A Saddler, Bruno Silva, Hamid Bagheri, Myra B Cohen, and Shane Farritor. 2018. Modeling and testing a family of surgical robots: an experience report. In the 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE). ACM, 785--790.
[32]
Ke Mao, Mark Harman, and Yue Jia. 2017. Crowd intelligence enhances automated mobile testing. In the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 16--26.
[33]
Ke Mao, Mark Harman, and Yue Jia. 2017. Robotic testing of mobile apps for truly black-box automation. IEEE Software 34, 2 (2017), 11--16.
[34]
Claudio Menghi, Christos Tsigkanos, Patrizio Pelliccione, Carlo Ghezzi, and Thorsten Berger. 2019. Specification patterns for robotic missions. arXiv preprint arXiv:1901.02077 (2019).
[35]
Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of Android applications. In IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, 559--570.
[36]
Clemens Mühlbacher, Gerald Steinbauer, Michael Reip, and Stephan Gspandl. 2019. Constraint-based testing of an industrial multi-robot navigation system. In IEEE International Conference On Artificial Intelligence Testing. IEEE, 129--137.
[37]
Sinan Naji, Hamid A. Jalab, and Sameem A. Kareem. 2019. A survey on skin detection in colored images. Artificial Intelligence Review 52, 2 (2019), 1041--1087.
[38]
Shaul Oron, Tali Dekel, Tianfan Xue, William T Freeman, and Shai Avidan. 2017. Best-buddies similarity---Robust template matching using mutual nearest neighbors. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 8 (2017), 1799--1813.
[39]
Zhengrui Qin, Yutao Tang, Ed Novak, and Qun Li. 2016. MobiPlay: A remote execution based record-and-replay tool for mobile applications. In the 38th International Conference on Software Engineering (ICSE). ACM, 571--582.
[40]
Paridhi Swaroop and Neelam Sharma. 2016. An overview of various template matching methodologies in image processing. International Journal of Computer Applications 153, 10 (2016), 8--14.
[41]
Richard Szeliski. 2010. Computer vision: algorithms and applications (1st ed.). Springer-Verlag, Berlin, Heidelberg.
[42]
Ragnar Wernersson. 2015. Robot control and computer vision for automated test system on touch display products. Master's thesis. Lund University, Sweden.
[43]
Thomas D White, Gordon Fraser, and Guy J Brown. 2019. Improving random GUI testing with image-based widget detection. In 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, 307--317.
[44]
Rahulkrishna Yandrapally, Giriprasad Sridhara, and Saurabh Sinha. 2015. Automated modularization of GUI test cases. In Proceedings of the 37th International Conference on Software Engineering (ICSE). IEEE, 44--54.

Cited By

View all
  • (2025)Enhanced Crowdsourced Test Report Prioritization via Image-and-Text Semantic Understanding and Feature IntegrationIEEE Transactions on Software Engineering10.1109/TSE.2024.351637251:1(283-304)Online publication date: 1-Jan-2025
  • (2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
  • (2024)Effective, Platform-Independent GUI Testing via Image Embedding and Reinforcement LearningACM Transactions on Software Engineering and Methodology10.1145/367472833:7(1-27)Online publication date: 21-Jun-2024
  • Show More Cited By

Index Terms

  1. RoScript: a visual script driven truly non-intrusive robotic testing system for touch screen applications

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering
    June 2020
    1640 pages
    ISBN:9781450371216
    DOI:10.1145/3377811
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    In-Cooperation

    • KIISE: Korean Institute of Information Scientists and Engineers
    • IEEE CS

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 01 October 2020

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. GUI testing
    2. computer vision
    3. non-intrusive
    4. robot
    5. test automation

    Qualifiers

    • Research-article

    Funding Sources

    • National Key R&D Program of China
    • Equipment Pre-Research Project of China
    • National Natural Science Foundation of China

    Conference

    ICSE '20
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 276 of 1,856 submissions, 15%

    Upcoming Conference

    ICSE 2025

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)54
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 18 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Enhanced Crowdsourced Test Report Prioritization via Image-and-Text Semantic Understanding and Feature IntegrationIEEE Transactions on Software Engineering10.1109/TSE.2024.351637251:1(283-304)Online publication date: 1-Jan-2025
    • (2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
    • (2024)Effective, Platform-Independent GUI Testing via Image Embedding and Reinforcement LearningACM Transactions on Software Engineering and Methodology10.1145/367472833:7(1-27)Online publication date: 21-Jun-2024
    • (2024)RFNIT: Robotic Framework for Non-invasive TestingCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663871(667-668)Online publication date: 10-Jul-2024
    • (2024)Semi-supervised Crowdsourced Test Report Clustering via Screenshot-Text Binding RulesProceedings of the ACM on Software Engineering10.1145/36607761:FSE(1540-1563)Online publication date: 12-Jul-2024
    • (2024)On Improving Management of Duplicate Video-Based Bug ReportsProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3639786(201-203)Online publication date: 14-Apr-2024
    • (2024)Practical Non-Intrusive GUI Exploration Testing with Visual-based Robotic ArmsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639161(1-13)Online publication date: 20-May-2024
    • (2024)Robotic Process Automation Efficiency for Mobile App Testing: An Empirical InvestigationInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402450011634:07(1025-1046)Online publication date: 14-May-2024
    • (2024)A real-time object detection method for electronic screen GUI test systemsThe Journal of Supercomputing10.1007/s11227-024-06319-y80:15(22803-22835)Online publication date: 1-Jul-2024
    • (2024)Semantic matching in GUI test reuseEmpirical Software Engineering10.1007/s10664-023-10406-829:3Online publication date: 9-May-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media