research-article

Feature Matching-based Approaches to Improve the Robustness of Android Visual GUI Testing

Authors:
Luca Ardito

Politecnico di Torino, Torino, Italy

Politecnico di Torino, Torino, Italy
View Profile

,
Andrea Bottino

Politecnico di Torino, Torino, Italy

Politecnico di Torino, Torino, Italy
View Profile

,
Riccardo Coppola

Politecnico di Torino, Torino, Italy

Politecnico di Torino, Torino, Italy
View Profile

,
Fabrizio Lamberti

Politecnico di Torino, Torino, Italy

Politecnico di Torino, Torino, Italy
View Profile

,
Francesco Manigrasso

Politecnico di Torino, Torino, Italy

Politecnico di Torino, Torino, Italy
View Profile

,
Lia Morra

Politecnico di Torino, Torino, Italy

Politecnico di Torino, Torino, Italy
View Profile

,
Marco Torchiano

Politecnico di Torino, Torino, Italy

Politecnico di Torino, Torino, Italy
View Profile

ACM Transactions on Software Engineering and Methodology Volume 31 Issue 2Article No.: 21pp 1–32https://doi.org/10.1145/3477427

Published:17 November 2021Publication History

ACM Transactions on Software Engineering and Methodology

Abstract

In automated Visual GUI Testing (VGT) for Android devices, the available tools often suffer from low robustness to mobile fragmentation, leading to incorrect results when running the same tests on different devices.

To soften these issues, we evaluate two feature matching-based approaches for widget detection in VGT scripts, which use, respectively, the complete full-screen snapshot of the application (Fullscreen) and the cropped images of its widgets (Cropped) as visual locators to match on emulated devices.

Our analysis includes validating the portability of different feature-based visual locators over various apps and devices and evaluating their robustness in terms of cross-device portability and correctly executed interactions. We assessed our results through a comparison with two state-of-the-art tools, EyeAutomate and Sikuli.

Despite a limited increase in the computational burden, our Fullscreen approach outperformed state-of-the-art tools in terms of correctly identified locators across a wide range of devices and led to a 30% increase in passing tests.

Our work shows that VGT tools’ dependability can be improved by bridging the testing and computer vision communities. This connection enables the design of algorithms targeted to domain-specific needs and thus inherently more usable and robust.

REFERENCES

[1] AB Synteda. 2018. EyeAutomate Documentation. Retrieved from https://eyeautomate.com/wp-content/themes/EyeAutomateTheme/resources/EyeAutomateCertifiedTesterCourse.pdf.Google Scholar
[2] Alégroth Emil. 2015. Visual GUI Testing: Automating High-level Software Testing in Industrial Practice. Chalmers University of Technology, Göteborg.Google Scholar
[3] Alégroth Emil and Feldt Robert. 2017. On the long-term use of visual GUI testing in industrial practice: A case study. Empir. Softw. Eng. 22, 6 (2017), 2937–2971. Google ScholarDigital Library
[4] Alégroth Emil, Feldt Robert, and Kolström Pirjo. 2016. Maintenance of automated test suites in industry: An empirical study on visual GUI testing. Info. Softw. Technol. 73 (2016), 66–80. Google ScholarDigital Library
[5] Alégroth Emil, Gao Zebao, Oliveira Rafael, and Memon Atif. 2015. Conceptualization and evaluation of component-based testing unified with visual GUI testing: An empirical study. In Proceedings of the IEEE 8th International Conference on Software Testing, Verification and Validation (ICST’15). IEEE, 1–10.Google ScholarCross Ref
[6] Alegroth Emil, Nass Michel, and Olsson Helena H.. 2013. JAutomate: A tool for system-and acceptance-test automation. In Proceedings of the IEEE 6th International Conference on Software Testing, Verification and Validation. IEEE, 439–446. Google ScholarDigital Library
[7] Amalfitano Domenico, Fasolino Anna Rita, Tramontana Porfirio, Ta Bryan Dzung, and Memon Atif M. 2015. MobiGUITAR: Automated model-based testing of mobile apps. IEEE Softw. 32, 5 (2015), 53–59.Google ScholarDigital Library
[8] Amalfitano Domenico, Riccio Vincenzo, Paiva Ana C. R., and Fasolino Anna Rita. 2018. Why does the orientation change mess up my Android application? From GUI failures to code faults. Softw. Test. Verificat. Reliabil. 28, 1 (2018), e1654.Google ScholarCross Ref
[9] Anderson Marti J.. 2014. Permutational multivariate analysis of variance (PERMANOVA). Wiley Statsref: Stat. Ref. Online 1 (2014), 1–15.Google Scholar
[10] Ardito Luca, Coppola Riccardo, Morisio Maurizio, and Torchiano Marco. 2019. Espresso vs. EyeAutomate: An experiment for the comparison of two generations of android GUI testing. In Proceedings of the Evaluation and Assessment on Software Engineering. ACM, 13–22. Google ScholarDigital Library
[11] Ardito Luca, Coppola Riccardo, Torchiano Marco, and Alégroth Emil. 2018. Towards automated translation between generations of GUI-based tests for mobile devices. In Companion Proceedings for the ISSTA/ECOOP Workshops. ACM, 46–53. Google ScholarDigital Library
[12] Banerjee Ishan, Nguyen Bao, Garousi Vahid, and Memon Atif. 2013. Graphical user interface (GUI) testing: Systematic mapping and repository. Info. Softw. Technol. 55, 10 (2013), 1679–1694. Google ScholarDigital Library
[13] Barath Daniel and Matas Jiri. 2017. Graph-Cut RANSAC. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 6733–6741.Google Scholar
[14] Bay Herbert, Ess Andreas, Tuytelaars Tinne, and Gool Luc Van. 2008. Speeded-up robust features (SURF). Comput. Vis. Image Underst. 110, 3 (June 2008), 346–359. https://doi.org/10.1016/j.cviu.2007.09.014 Google ScholarDigital Library
[15] Borges Nataniel P., Gómez Maria, and Zeller Andreas. 2018. Guiding app testing with mined interaction models. In Proceedings of the 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft’18). Association for Computing Machinery, New York, NY, 133–143. https://doi.org/10.1145/3197231.3197243 Google ScholarDigital Library
[16] Borjesson Emil and Feldt Robert. 2012. Automated system testing using visual GUI testing tools: A comparative study in industry. In Proceedings of the IEEE 5th International Conference on Software Testing, Verification and Validation. IEEE, 350–359. Google ScholarDigital Library
[17] Bradski G.. 2000. The OpenCV library. Dr. Dobb’s J. Softw. Tools 1 (2000).Google Scholar
[18] Caldiera Victor R. Basili-Gianluigi and Rombach H. Dieter. 1994. Goal question metric paradigm. Encyclopedia Softw. Eng. 1 (1994), 528–532.Google Scholar
[19] Chang Tsung-Hsiang, Yeh Tom, and Miller Robert C.. 2010. GUI testing using computer vision. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, 1535–1544. Google ScholarDigital Library
[20] Chen Jieshan, Xie Mulong, Xing Zhenchang, Chen Chunyang, Xu Xiwei, Zhu Liming, and Li Guoqiang. 2020. Object detection for graphical user interface: Old fashioned or deep learning or a combination? In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for Computing Machinery, New York, NY, 1202–1214. Google ScholarDigital Library
[21] Coppola Riccardo, Ardito Luca, and Torchiano Marco. 2019. Fragility of layout-based and visual GUI test scripts: An assessment study on a hybrid mobile application. In Proceedings of the 10th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. Association for Computing Machinery, New York, NY, 28–34. Google ScholarDigital Library
[22] Coppola Riccardo, Ardito Luca, Torchiano Marco, and Alégroth Emil. 2020. Translation from visual to layout-based android test cases: A proof of concept. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW’20). IEEE, Washington, DC, 74–83.Google ScholarCross Ref
[23] Coppola Riccardo, Ardito Luca, Torchiano Marco, and Alégroth Emil. 2021. Translation from layout-based to visual android test scripts: An empirical evaluation. J. Syst. Softw. 171 (2021), 110845. https://doi.org/10.1016/j.jss.2020.110845Google ScholarCross Ref
[24] Coppola Riccardo, Morisio Maurizio, Torchiano Marco, and Ardito Luca. 2019. Scripted GUI testing of Android open-source apps: Evolution of test code and fragility causes. Empir. Softw. Eng. 24 (2019), 1–44.Google ScholarDigital Library
[25] Datta Ritendra, Joshi Dhiraj, Li Jia, and Wang James Z.. 2008. Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. 40, 2, Article 5 (May 2008), 60 pages. https://doi.org/10.1145/1348246.1348248 Google ScholarDigital Library
[26] Deka Biplab, Huang Zifeng, Franzen Chad, Hibschman Joshua, Afergan Daniel, Li Yang, Nichols Jeffrey, and Kumar Ranjitha. 2017. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, 845–854. Google ScholarDigital Library
[27] Developers A.. 2012. Ui/application exerciser monkey.Google Scholar
[28] Fazzini Mattia, Freitas Eduardo Noronha de A., Choudhary Shauvik Roy, and Orso Alessandro. 2017. Barista: A technique for recording, encoding, and running platform independent android tests. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST’17). IEEE, Washington, DC, 149–160.Google ScholarCross Ref
[29] Alcantarilla Pablo Fernández. 2013. Fast explicit diffusion for accelerated features in nonlinear scale spaces. In Proceedings of the British Machine Vision Conference. BMVA Press, Durham, UK. https://doi.org/10.5244/C.27.13Google ScholarCross Ref
[30] Fukunaga K. and Hostetler L.. 1975. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans. Info. Theory 21, 1 (1975), 32–40. Google ScholarDigital Library
[31] Han Dan, Zhang Chenlei, Fan Xiaochao, Hindle Abram, Wong Kenny, and Stroulia Eleni. 2012. Understanding android fragmentation with topic analysis of vendor-specific bugs. In Proceedings of the 19th Working Conference on Reverse Engineering (WCRE’12). IEEE, Washington, DC, 83–92. Google ScholarDigital Library
[32] Hasselknippe Kristian Fjeld and Li Jingyue. 2017. A novel tool for automatic GUI layout testing. In Proceedings of the 24th Asia-Pacific Software Engineering Conference (APSEC’17). IEEE, Washington, DC, 695–700.Google ScholarCross Ref
[33] Hu Jiajun, Wei Lili, Liu Yepang, Cheung Shing-Chi, and Huang Huaxun. 2018. A tale of two cities: How webview induces bugs to android applications. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. Association for Computing Machinery, New York, NY, 702–713. Google ScholarDigital Library
[34] Kaasila Jouko, Ferreira Denzil, Kostakos Vassilis, and Ojala Timo. 2012. Testdroid: Automated remote UI testing on Android. In Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia. Association for Computing Machinery, New York, NY, 1–4. Google ScholarDigital Library
[35] Kowalczyk Emily, Cohen Myra B., and Memon Atif M.. 2018. Configurations in Android testing: They matter. In Proceedings of the 1st International Workshop on Advances in Mobile App Analysis. Association for Computing Machinery, New York, NY, 1–6. Google ScholarDigital Library
[36] Leotta Maurizio, Stocco Andrea, Ricca Filippo, and Tonella Paolo. 2018. Pesto: Automated migration of DOM-based Web tests towards the visual approach. Softw. Test. Verificat. Reliabil. 28, 4 (2018), e1665.Google ScholarCross Ref
[37] Lin Ying-Dar, Rojas Jose F., Chu Edward T.-H., and Lai Yuan-Cheng. 2014. On the accuracy, efficiency, and reusability of automated test oracles for android devices. IEEE Trans. Softw. Eng. 40, 10 (2014), 957–970.Google ScholarCross Ref
[38] Linares-Vásquez Mario, Moran Kevin, and Poshyvanyk Denys. 2017. Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing. In Proceedings of the IEEE International Conference on. IEEE, Washington, DC, 399–410.Google ScholarCross Ref
[39] Lowe David G.. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 2 (2004), 91–110. Google ScholarDigital Library
[40] Luna Eduardo and Ariss Omar El. 2018. Edroid: A mutation tool for android apps. In Proceedings of the 6th International Conference in Software Engineering Research and Innovation (CONISOFT’18). IEEE, Washington, DC, 99–108.Google ScholarCross Ref
[41] Mangiafico S. S.. 2015. An R Companion for the Handbook of Biological Statistics, Version 1.09c, 274 pp.Google Scholar
[42] Mao Ke, Harman Mark, and Jia Yue. 2016. Sapienz: Multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis. Association for Computing Machinery, New York, NY, 94–105. Google ScholarDigital Library
[43] Martinez Matias and Mateus Bruno Gois. 2020. How and Why did developers migrate Android Applications from Java to Kotlin? A study based on code analysis and interviews with developers. arXiv:2003.12730v1 [cs.SE].Google Scholar
[44] Merlo Alessio and Georgiu Gabriel Claudiu. 2017. Riskindroid: Machine learning-based risk analysis on android. In Proceedings of the IFIP International Conference on ICT Systems Security and Privacy Protection. Springer, Cham, 538–552.Google ScholarCross Ref
[45] Morales-Ortega Salvador, Escamilla-Ambrosio Ponciano Jorge, Rodriguez-Mota Abraham, and Coronado-De-Alba Lilian D.. 2016. Native malware detection in smartphones with Android OS using static analysis, feature selection and ensemble classifiers. In Proceedings of the 11th International Conference on Malicious and Unwanted Software (MALWARE’16). IEEE, Washington, DC, 1–8.Google ScholarCross Ref
[46] Moran Kevin, Bonett Richard, Bernal-Cárdenas Carlos, Otten Brendan, Park Daniel, and Poshyvanyk Denys. 2017. On-device bug reporting for android applications. In Proceedings of the IEEE/ACM 4th International Conference on Mobile Software Engineering and Systems (MOBILESoft’17). IEEE, Washington, DC, 215–216. Google ScholarDigital Library
[47] Mozgovoy Maxim and Pyshkin Evgeny. 2017. Using image recognition for testing hand-drawn graphic user interfaces. In Proceedings of the the 11th International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies. IARIA, Barcelona, 25–28.Google Scholar
[48] Mozgovoy Maxim and Pyshkin Evgeny. 2018. Pragmatic approach to automated testing of mobile applications with non-native graphic user interface. In International Journal on Advances in Software. IARIA, Wilmington, UK, 239–246.Google Scholar
[49] Mozgovoy Maxim and Pyshkin Evgeny. 2018. Unity Application Testing Automation with Appium and Image Recognition. Springer, Cham, 139–150. https://doi.org/10.1007/978-3-319-71734-0_12Google Scholar
[50] Negara Stas, Esfahani Naeem, and Buse Raymond P. L.. 2019. Practical android test recording with espresso test recorder. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice. IEEE, Washington, DC, 193–202. Google ScholarDigital Library
[51] Park Je-Ho, Park Young Bom, and Ham Hyung Kil. 2013. Fragmentation problem in Android. In Proceedings of the International Conference on Information Science and Applications (ICISA’13). IEEE, Washington, DC, 1–2.Google ScholarCross Ref
[52] Shirazi Alireza Sahami, Henze Niels, Schmidt Albrecht, Goldberg Robin, Schmidt Benjamin, and Schmauder Hansjörg. 2013. Insights into layout patterns of mobile user interfaces by an automatic analysis of android apps. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems. Association for Computing Machinery, New York, NY, 275–284. Google ScholarDigital Library
[53] Statista. 2021. Global Google Play app downloads 2016-2019. Retrieved from https://www.statista.com/statistics/734332/google-play-app-installs-per-year/.Google Scholar
[54] Tareen S. A. K. and Saleem Z.. 2018. A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK. In Proceedings of the International Conference on Computing, Mathematics and Engineering Technologies (iCoMET’18). IEEE, Washington, DC, 1–10.Google ScholarCross Ref
[55] Tareen Shaharyar Ahmed Khan and Saleem Zahra. 2018. A comparative analysis of sift, surf, kaze, akaze, orb, and brisk. In Proceedings of the International conference on Computing, Mathematics and Engineering Technologies (iCoMET’18). IEEE, Washington, DC, 1–10.Google ScholarCross Ref
[56] Thorve Swapna, Sreshtha Chandani, and Meng Na. 2018. An empirical study of flaky tests in android apps. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME’18). IEEE, Washington, DC, 534–538.Google ScholarCross Ref
[57] Torchiano Marco. 2020. effsize: Efficient Effect Size Computation. Politecnico di Torino. https://doi.org/10.5281/zenodo.1480624Google Scholar
[58] Tuovenen J., Oussalah Mourad, and Kostakos Panos. 2019. MAuto: Automatic mobile game testing tool using image-matching based approach. Comput. Games J. 8 (Oct. 2019), 215–239. https://doi.org/10.1007/s40869-019-00087-zGoogle ScholarCross Ref
[59] Tuytelaars Tinne and Mikolajczyk Krystian. 2008. Local Invariant Feature Detectors: A Survey. Now Publishers, Boston, MA. Google ScholarDigital Library
[60] Vesikkala Mikko. 2014-05-05. Visual Regression Testing for Web Applications; Selainpohjaisten ohjelmistojen visuaalinen regressiotestaus. G2 Pro gradu, diplomityö; masterThesis. Aalto University. Retrieved from http://urn.fi/URN:NBN:fi:aalto-201405131809.Google Scholar
[61] Yan Jiwei, Liu Hao, Pan Linjie, Yan Jun, Zhang Jian, and Liang Bin. 2020. Multiple-entry testing of android applications by constructing activity launching contexts. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering (ICSE’20). IEEE, Washington, DC, 457–468. Google ScholarDigital Library
[62] Yeh Tom, Chang Tsung-Hsiang, and Miller Robert C.. 2009. Sikuli: Using GUI screenshots for search and automation. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, 183–192. Google ScholarDigital Library
[63] Zadgaonkar Hrushikesh. 2013. Robotium Automated Testing for Android. Packt Publishing, Birmingham, UK. Google ScholarDigital Library
[64] Zelenchuk Denys. 2019. Espresso and UI automator: The perfect tandem. In Android Espresso Revealed. Springer, Cham, 165–189.Google ScholarCross Ref

Index Terms

Feature Matching-based Approaches to Improve the Robustness of Android Visual GUI Testing
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Matching
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

A black-box based android GUI testing system
MobiSys '13: Proceeding of the 11th annual international conference on Mobile systems, applications, and services

In Android system, black box testing has risen to three key issues: S1: There is no source code and for tester to know the internal logic of the testing App. S2: There is no testing criterion for tester to know the correct behavior and testing scope of ...
Read More
Comparing automated visual GUI testing tools: an industrial case study
A-TEST 2017: Proceedings of the 8th ACM SIGSOFT International Workshop on Automated Software Testing

Visual GUI testing (VGT) is a tool-driven technique, which uses image recognition for interaction and assertion of the behaviour of system under test. Motivated by a real industrial need, in the context of a large Turkish software and systems company ...
Read More
Visual GUI testing in practice: challenges, problemsand limitations

In today's software development industry, high-level tests such as Graphical User Interface (GUI) based system and acceptance tests are mostly performed with manual practices that are often costly, tedious and error prone. Test automation has been ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Transactions on Software Engineering and Methodology Volume 31, Issue 2
April 2022
789 pages
ISSN:1049-331X
EISSN:1557-7392
DOI:10.1145/3492439
Editor:
Mauro Pezzè
USI Università della Svizzera italiana and SIT Schaffhausen Institute of Technology, Switzerland
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 17 November 2021
- Revised: 1 July 2021
- Accepted: 1 July 2021
- Received: 1 September 2020
Published in tosem Volume 31, Issue 2

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
Mobile computing
software testing
visual GUI testing
feature matching
Qualifiers
- research-article
- Refereed
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 3
  Total Citations
  View Citations
- 717
  Total Downloads
- Downloads (Last 12 months)196
- Downloads (Last 6 weeks)34
Other Metrics
View Author Metrics
Cited By
View all

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

View Full Text

HTML Format

View this article in HTML Format .

View HTML Format

Feature Matching-based Approaches to Improve the Robustness of Android Visual GUI Testing

ACM Transactions on Software Engineering and Methodology

Abstract

REFERENCES

Cited By

Index Terms

Recommendations

A black-box based android GUI testing system

Comparing automated visual GUI testing tools: an industrial case study

Visual GUI testing in practice: challenges, problemsand limitations