skip to main content
10.1145/3507524.3507541acmotherconferencesArticle/Chapter ViewAbstractPublication PagesiccbdConference Proceedingsconference-collections
research-article

Pic2Translator: A tool to extract mobile GUI components for machine learning and tasks

Published: 08 March 2022 Publication History

Abstract

Mobile apps are becoming more popular in society whether professional, entertainment or simply communication fields since they can provide a collection of useful features in daily activities. One of the essential requirements for an application to be successful in the market is the Graphical User Interface which encompasses most of the user interaction. Thus, the GUI test is an important practice to control the issues that affect the interaction and user experience. However, those tests face some challenges like high-cost maintenance in UI changes. This paper presents a tool that can extract information from mobile GUI interfaces. The information extracted includes the GUI components type, layout hierarchy, and belonging scenario, as well as component position in the screen, component image, and pixel map. Such a pixel map provides the component class of each image pixel to enable semantic segmentation training schemes. Such a dataset may support the research on machine applications associated, for instance, with the identification of changes employing the user interface.

Supplemental Material

PPTX File - Pic2Translator
Presentation slides

References

[1]
Imparato, Gennaro. “A combined technique of GUI ripping and input perturbation testing for Android apps.” In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol 2, pp. 760-762. IEEE, 2015.
[2]
Grano, Giovanni, Adelina Ciurumelea, Sebastiano Panichella, Fabio Palomba, and Harald C. Gall. “Exploring the integration of user feedback in automated testing of android applications.” In 2018 IEEE 25Th international conference on software analysis, evolution and reengineering (SANER), pp. 72-83. IEEE, 2018.
[3]
“Mobile App Growth Statistics in 2021”. In https://www;appventurez.com/blog/mobile-app-statistics/
[4]
Chen, Sen, Lingling Fan, Ting Su, Lei Ma, Yang Liu, and Lihua Xu. “Automated cross-platform GUI code generation for mobile apps.” In 2019 IEEE 1st International Workshop on Artificial Intelligence for Mobile (AI4Mobile), pp. 13-16. IEEE, 2019.
[5]
Chen, Sen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, and Lihua Xu. “Storydroid: Automated generation of storyboard for Android apps.” In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 596-607. IEEE, 2019.
[6]
Chang, Nana, Linzhang Wang, Yu Pei, Subrota K. Mondal, and Xuandong Li. “Change-based test script maintenance for android apps.” In 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS), pp. 215-225. IEEE, 2018.
[7]
Coppola, Riccardo, Maurizio Morisio, and Marco Torchiano. “Mobile gui testing fragility: A study on open-source android applications.” IEEE Transactions on Reliability 68, no. 1 (2018): 67-90.
[8]
Pan, Minxue, Tongtong Xu, Yu Pei, Zhong Li, Tian Zhang, and Xuandong Li. "GUI-Guided Test Script Repair for Mobile Apps." IEEE Transactions on Software Engineering (2020).
[9]
Linares-Vásquez, Mario, Gabriele Bavota, Carlos Bernal-Cárdenas, Massimiliano Di Penta, Rocco Oliveto, and Denys Poshyvanyk. "Api change and fault proneness: A threat to the success of android apps." In Proceedings of the 2013 9th joint meeting on foundations of software engineering, pp. 477-487. 2013.
[10]
Hu, Gang, Xinhao Yuan, Yang Tang, and Junfeng Yang. "Efficiently, effectively detecting mobile app bugs with appdoctor." In Proceedings of the Ninth European Conference on Computer Systems, pp. 1-15. 2014.
[11]
Moran, Kevin, Cody Watson, John Hoskins, George Purnell, and Denys Poshyvanyk. "Detecting and summarizing GUI changes in evolving mobile apps." In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 543-553. 2018.
[12]
Coppola, Riccardo, Maurizio Morisio, and Marco Torchiano. "Maintenance of Android Widget-based GUI Testing: A Taxonomy of test case modification causes." In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 151-158. IEEE, 2018.
[13]
Grechanik, Mark, Qing Xie, and Chen Fu. "Experimental assessment of manual versus tool-based maintenance of GUI-directed test scripts." In 2009 IEEE International Conference on Software Maintenance, pp. 9-18. IEEE, 2009.
[14]
Lin, Jun-Wei, Reyhaneh Jabbarvand, and Sam Malek. "Test transfer across mobile apps through semantic mapping." In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 42-53. IEEE, 2019.
[15]
Linares-Vásquez, Mario, Martin White, Carlos Bernal-Cárdenas, Kevin Moran, and Denys Poshyvanyk. "Mining android app usages for generating actionable gui-based execution scenarios." In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, pp. 111-122. IEEE, 2015.
[16]
Deka, Biplab, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. "Rico: A mobile app dataset for building data-driven design applications." In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845-854. 2017.
[17]
Beltramelli, Tony. "pix2code: Generating code from a graphical user interface screenshot." In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 1-6. 2018.
[18]
Nguyen, Tam, Phong Vu, Hung Pham, and Tung Nguyen. "Deep learning UI design patterns of mobile apps." In 2018 IEEE/ACM 40th International Conference on Software Engineering: New Ideas and Emerging Technologies Results (ICSE-NIER), pp. 65-68. IEEE, 2018.
[19]
Moran, Kevin, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. "Machine learning-based prototyping of graphical user interfaces for mobile apps." IEEE Transactions on Software Engineering 46, no. 2 (2018): 196-221.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICCBD '21: Proceedings of the 2021 4th International Conference on Computing and Big Data
November 2021
148 pages
ISBN:9781450387194
DOI:10.1145/3507524
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 March 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. graphical user interface
  2. mobile
  3. semantic segmentation
  4. training dataset

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICCBD 2021

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 44
    Total Downloads
  • Downloads (Last 12 months)8
  • Downloads (Last 6 weeks)2
Reflects downloads up to 13 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media