Abstract
In mobile application development, building a consistent user interface (UI) might be a costly and time-consuming process. This is especially the case if an organization has a separate team for each mobile platform such as iOS and Android. In this regard, the companies that choose the native mobile app development path end up going through do-overs as the UI work done on one platform needs to be repeated for other platforms too. One of the tedious parts of UI design tasks is creating a graphical user interface (GUI). There are numerous tools and prototypes in the literature that aim to create feasible GUI automation solutions to speed up this process and reduce the labor workload. However, as the technologies evolve and improve new versions of existing algorithms are created and offered. Accordingly, this study aims to employ the latest version of YOLO, which is YOLOv5, to create a custom object detection model that recognizes GUI elements in a given UI image. In order to benchmark the newly trained YOLOv5 GUI element detection model, existing work from the literature and their data set is considered and used for comparison purposes. Therefore, this study makes use of 450 UI samples of the VINS dataset for testing, a similar amount for validation and the rest for model training. Then the findings of this work are compared with another study that has used the SSD algorithm and VINS dataset to train, validate and test its model, which showed that proposed algorithm outperformed SSD’s mean average precision (mAP) by 15.69%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Puiu, T.: Your smartphone is millions of times more powerful than the Apollo 11 guidance computers. ZME Science (2021). https://www.zmescience.com/science/news-science/smartphone-power-compared-to-apollo-432/. Accessed 15 Jan 2022
Ceci, L.: Number of apps available in leading app stores as of 1st quarter 2021. Statista (2022). https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/. Accessed 30 Jan 2022
Statista Research Department: Revenue of mobile apps worldwide 2017–2025, by segment. Statista (2021). https://www.statista.com/statistics/269025/worldwide-mobile-app-revenue-forecast/. Accessed 02 Feb 2022
Chen, J., et al.: Object detection for graphical user interface: old fashioned or deep learning or a combination? In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1202–1214. ACM (2020)
Xie, M., et al.: UIED: a hybrid tool for GUI element detection. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1655–1659. ACM (2020)
Bunian, S., et al.: VINS: visual search for mobile user interface design. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–14. ACM (2021)
Nguyen, T.A., Csallner, C.: Reverse engineering mobile application user interfaces with REMAUI (T). In: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering, pp. 248–259. ACM (2015)
Chen, S., et al.: Automated cross-platform GUI code generation for mobile apps. In: IEEE 1st International Workshop on Artificial Intelligence for Mobile, pp. 13–16. IEEE (2019)
OpenCV About. https://opencv.org/about/. Accessed 13 Feb 2022
Apple Developer Vision. https://developer.apple.com/documentation/vision. Accessed 16 Feb 2022
Google Colab. https://colab.research.google.com. Accessed 1 Jan 2022
Zhiqiang, W., Jun, L.: A review of object detection based on convolutional neural network. In: Proceedings of the 36th Chinese Control Conference, pp. 85–112. IEEE (2017)
Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448. IEEE (2015)
Redmon, J., et al.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788. IEEE (2016)
Deka, B., et al.: Rico: a mobile app dataset for building data-driven design applications. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845–854. ACM (2017)
What is Uplabs? https://www.uplabs.com/faq. Accessed 18 Feb 2022
Lin, T..-Y.., et al.: Microsoft coco: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Unel, F.O., et al.: The power of tiling for small object detection. In: Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 582–591. IEEE (2019)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Altinbas, M.D., Serif, T. (2022). GUI Element Detection from Mobile UI Images Using YOLOv5. In: Awan, I., Younas, M., Poniszewska-Marańda, A. (eds) Mobile Web and Intelligent Information Systems. MobiWIS 2022. Lecture Notes in Computer Science, vol 13475. Springer, Cham. https://doi.org/10.1007/978-3-031-14391-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-14391-5_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14390-8
Online ISBN: 978-3-031-14391-5
eBook Packages: Computer ScienceComputer Science (R0)