Skip to main content
SpringerLink
Log in

Detection and Segmentation of Graphical Elements on GUIs for Mobile Apps Based on Deep Learning

  • Conference paper
  • First Online:
Mobile Computing, Applications, and Services (MobiCASE 2020)

Abstract

Recently, mobile devices are more popular than computers. However, mobile apps are not as thoroughly tested as desktop ones, especially for graphical user interface (GUI). In this paper, we study the detection and segmentation of graphical elements on GUIs for mobile apps based on deep learning. It is the preliminary work of GUI testing for mobile apps based on artificial intelligence. We create a dataset, which consists of 2,100 GUI screenshots (or pages) labeled with 42,156 graphic elements in 8 classes. Based on our dataset, we adopt Mask R-CNN to train the detection and segmentation of graphic elements on GUI screenshots. The experimental results show that the mAP value achieves 98%.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Moran, K., Linares-Vasquez, M., Bernal-Cardenas, C.: Automatically discovering, reporting and reproducing android application crashes. In: ICST, pp. 33–44. IEEE Computer Society, Los Alamitos (2016)

    Google Scholar 

  2. Khalid, H., Shihab, E., Nagappan, M.: What do mobile app users complain about? IEEE Softw. 32(3), 70–77 (2015)

    Article  Google Scholar 

  3. Kaur, A.: Review of mobile applications testing with automated techniques. Int. J. Adv. Res. Comput. Commun. Eng. 4(10), 503–507 (2015)

    Google Scholar 

  4. Coppola, R., Raffero, E., Torchiano, M.: Automated mobile UI test fragility: an exploratory assessment study on Android. In: INTUITEST 2016: Proceedings of the 2nd International Workshop on User Interface Test Automation, pp. 11–20. ACM, New York (2016)

    Google Scholar 

  5. Muccini, H., Francesco, A.D., Esposito, P.: Software testing of mobile applications: challenges and future research directions. In: International Workshop on Automation of Software Test (AST), pp. 29–35. IEEE Computer Society, Los Alamitos (2012)

    Google Scholar 

  6. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR, pp. 580–587. IEEE Computer Society, Los Alamitos (2014)

    Google Scholar 

  7. Deka, B., Huang, Z., Franzen, C.: Rico: a mobile app dataset for building data-driven design applications. In: UIST ‘17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845–854. ACM, New York (2017)

    Google Scholar 

  8. Choudhary, S., Gorla, A., Orso, A.: Automated test input generation for android: are we there yet? In: 30th IEEE/ACM International Conference on Automated Software Engineering, pp. 429–440. IEEE Computer Society, Los Alamitos (2015)

    Google Scholar 

  9. UI/Application Exerciser Monkey. https://developer.android.com/studio/test/monkey.html Accessed 27 July 2020

  10. Machiry, A., Tahiliani, R., Naik, M.: Dynodroid: an input generation system for android apps. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 224–234. ACM, New York (2013)

    Google Scholar 

  11. Amalfitano, D., Fasolino, A., Tramontana, P., Ta, B., Memon, A.: MobiGUITAR: automated model-based testing of mobile apps. IEEE Softw. 32(5), 53–59 (2015)

    Article  Google Scholar 

  12. Yang, W., Prasad, M.R., Xie, T.: A grey-box approach for automated GUI-model generation of mobile applications. In: Cortellessa, V., Varró, D. (eds.) FASE 2013. LNCS, vol. 7793, pp. 250–265. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37057-1_19

    Chapter  Google Scholar 

  13. Amalfitano, D., Fasolino, A., Tramontana, P., De Carmine, S., Memon, A.: Using GUI ripping for automated testing of android applications. In: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, pp. 258–261. ACM, New York (2012)

    Google Scholar 

  14. Azim, T., Neamtiu, I.: Targeted and depth-first exploration for systematic testing of android apps. In: Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications, pp. 641–660. ACM, New York (2013)

    Google Scholar 

  15. Bhoraskar, R., Han, S., Jeon, J.: Brahmastra: driving apps to test the security of third-party components. In: Proceedings of the 23rd USENIX Conference on Security Symposium, pp. 1021–1036. USENIX Association, San Diego (2014)

    Google Scholar 

  16. Pretschner, A., Prenninger, W., Wagner, S.: One evaluation of model-based testing and its automation. In: Proceedings of the 27th International Conference on Software Engineering, pp. 392–401. ACM, New York (2015)

    Google Scholar 

  17. Girshick, R.: Fast R-CNN. In: ICCV, pp. 1440–1448. IEEE Computer Society, Los Alamitos (2015)

    Google Scholar 

  18. Ren, S.Q., He, K.M., Girshick, R.B.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS, pp. 91–99. MIT Press, Cambridge (2015)

    Google Scholar 

  19. Redmon, J., Divvala, S., Girshick, R.: You only look once: unified, real-time object detection. In: CVPR, pp. 779–788. IEEE Computer Society, Los Alamitos (2016)

    Google Scholar 

  20. Liu, W., et al.: SSD: single shot multibox detector. In: Xie, T., Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  21. Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: CVPR, pp. 6517–6525. IEEE Computer Society, Los Alamitos (2017)

    Google Scholar 

  22. Wei, Z.Y., Wen, C., Xie, K.: Real-time face detection for mobile devices with optical flow estimation. J. Comput. Appl. 38(4), 1146–1150 (2018)

    Google Scholar 

  23. Li, J.W., Zhou, X.L., Chan, S.X.: A novel video target tracking method based on adaptive convolutional neural network feature. J. Comput. Aided Des. Comput. Graph. 30(2), 273–281 (2018)

    Google Scholar 

  24. He, K.M., Gkioxari, G., Dollar, P.: Mask R-CNN. In: ICCV, pp. 2980–2988. IEEE Computer Society, Los Alamitos (2017)

    Google Scholar 

  25. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv.org, https://arxiv.org/abs/1409.1556 Accessed 3 July 2020

  26. He, K.M., Zhang, X.Y., Ren, S.Q.: Deep residual learning for image recognition. In: CVPR, pp. 770–778. IEEE Computer Society, Los Alamitos (2016)

    Google Scholar 

Download references

Acknowledgement

This work is funded by National Key R&D Program of China (No. 2018YFB1403400), and Science and Technology Commission of Shanghai Municipality Program, China. (Nos. 17411952800, 18DZ2203700, 18DZ1113400).

Author information

Authors and Affiliations

  1. Shanghai Key Laboratory of Computer Software Testing and Evaluating, Shanghai Development Center of Computer Software Technology, Shanghai, China

    Rui Hu, Mingang Chen, Lizhi Cai & Wenjie Chen

Authors
  1. Rui Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lizhi Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenjie Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingang Chen .

Editor information

Editors and Affiliations

  1. East China Normal University, Shanghai, China

    Jing Liu

  2. Shanghai University, Shanghai, China

    Honghao Gao

  3. College of Computer, Hangzhou Dianzi University, Hangzhou, China

    Yuyu Yin

  4. Shanghai University of Electric Power, Shanghai, China

    Zhongqin Bi

Rights and permissions

Reprints and permissions

Copyright information

© 2020 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hu, R., Chen, M., Cai, L., Chen, W. (2020). Detection and Segmentation of Graphical Elements on GUIs for Mobile Apps Based on Deep Learning. In: Liu, J., Gao, H., Yin, Y., Bi, Z. (eds) Mobile Computing, Applications, and Services. MobiCASE 2020. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 341. Springer, Cham. https://doi.org/10.1007/978-3-030-64214-3_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-64214-3_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-64213-6

  • Online ISBN: 978-3-030-64214-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation