Skip to main content
SpringerLink
Log in

Localizing Non-functional Code Bugs in User Interfaces Using Deep Learning Techniques

  • Conference paper
  • First Online:
Model and Data Engineering (MEDI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14396))

Included in the following conference series:

  • 195 Accesses

Abstract

In the context of modern business digitization, non-functional code issues in User Interface (UI) exert a substantial impact on User Experience(UX). UX-related problems can generate annoyance and potentially result in revenue erosion. However, conventional approaches for detecting and resolving these defects tend to be prolonged, mandate a substantial degree of expertise, and divert precious time and resources away from vital software development tasks, ultimately impairing business operations. A deep learning-based approach automatically detects and localizes non-functional code flaws in user interface (UI) screens. The automated approach reduces the time and effort required to fix non-functional code bugs in UI screens, improving overall quality control and testing and decreasing overall UI testing time. Our model can identify non-functional code flaws by analyzing many UI screens and pinpointing their location on UI screens. We evaluate our proposed approach on a variety of UI screens. The proposed approach achieves better performance in classification by 4% and in localization by 16%.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Balakrishnan, R., Baudisch, P., Hinckley, K., Wigdor, D.: Rico: a mobile app dataset for building data-driven design applications. https://www.interactionmining.org/rico.html. Accessed 2 Feb 2023

  2. Crease, M., Brewster, S., Gray, P.: Caring, sharing widgets: a toolkit of sensitive widgets. In: McDonald, S., Waern, Y., Cockton, G. (eds.) People and Computers XIV–Usability or Else!, pp. 257–270. Springer, London (2000). https://doi.org/10.1007/978-1-4471-0515-2_17

    Chapter  Google Scholar 

  3. Deka, B., et al.: Rico: a mobile app dataset for building data-driven design applications. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST 2017, pp. 845–854. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3126594.3126651

  4. Denden, M., Tlili, A., Essalmi, F., Jemni, M.: Educational gamification based on personality. In: IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), pp. 1399–1405 (2017). https://doi.org/10.1109/AICCSA.2017.87

  5. Dyba, T., Dingsoyr, T., Hanssen, G.K.: Applying systematic reviews to diverse study types: an experience report. In: First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), pp. 225–234. IEEE (2007)

    Google Scholar 

  6. Fast, E., Steffee, D., Wang, L., Brandt, J.R., Bernstein, M.S.: Emergent, crowd-scale programming practice in the IDE. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2491–2500 (2014)

    Google Scholar 

  7. Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)

    Google Scholar 

  8. Goutte, C., Gaussier, E.: A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In: Losada, D.E., Fernández-Luna, J.M. (eds.) ECIR 2005. LNCS, vol. 3408, pp. 345–359. Springer, Heidelberg (2005). https://doi.org/10.1007/978-3-540-31865-1_25

    Chapter  Google Scholar 

  9. Haas, R., Elsner, D., Juergens, E., Pretschner, A., Apel, S.: How can manual testing processes be optimized? developer survey, optimization guidelines, and case studies. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1281–1291 (2021)

    Google Scholar 

  10. Harrison, R., Flood, D., Duce, D.: Usability of mobile applications: literature review and rationale for a new usability model. J. Interact. Sci. 1, 1–16 (2013)

    Article  Google Scholar 

  11. Hemmati, H., Fang, Z., Mantyla, M.V.: Prioritizing manual test cases in traditional and rapid release environments. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–10 (2015). https://doi.org/10.1109/ICST.2015.7102602

  12. Hemmati, H., Fang, Z., Mantyla, M.V.: Prioritizing manual test cases in traditional and rapid release environments. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–10. IEEE (2015)

    Google Scholar 

  13. Hemmati, H., Sharifi, F.: Investigating NLP-based approaches for predicting manual test case failure. In: 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), pp. 309–319. IEEE (2018)

    Google Scholar 

  14. Ines, G., Makram, S., Mabrouka, C., Mourad, A.: Evaluation of mobile interfaces as an optimization problem. Procedia Comput. Sci. 112, 235–248 (2017)

    Article  Google Scholar 

  15. Jansen, B.J.: The graphical user interface. ACM SIGCHI Bull. 30(2), 22–26 (1998)

    Article  Google Scholar 

  16. Joorabchi, M.E., Mesbah, A., Kruchten, P.: Real challenges in mobile app development. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 15–24. IEEE (2013)

    Google Scholar 

  17. Lee, M., Kent, T., Carswell, C.M., Seidelman, W., Sublette, M.: Zebra-striping: visual flow in grid-based graphic design. Proc. Hum. Factors Ergon. Soc. Ann. Meet. 58(1), 1318–1322 (2014). https://doi.org/10.1177/1541931214581275

    Article  Google Scholar 

  18. Li, Y., Bengio, S., Bailly, G.: Predicting human performance in vertical menu selection using deep learning. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–7 (2018)

    Google Scholar 

  19. Li, Y., Yang, Z., Guo, Y., Chen, X.: Humanoid: a deep learning-based approach to automated black-box android app testing. In: 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1070–1073. IEEE (2019)

    Google Scholar 

  20. Liu, Z., Chen, C., Wang, J., Huang, Y., Hu, J., Wang, Q.: Owl eyes: spotting UI display issues via visual understanding. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp. 398–409 (2020)

    Google Scholar 

  21. Liu, Z., Chen, C., Wang, J., Huang, Y., Hu, J., Wang, Q.: Guided bug crush: assist manual GUI testing of android apps via hint moves. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI 2022. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3491102.3501903

  22. Liu, Z., Chen, C., Wang, J., Huang, Y., Hu, J., Wang, Q.: Nighthawk: fully automated localizing UI display issues via visual understanding. IEEE Trans. Softw. Eng. 49(1), 403–418 (2022)

    Article  Google Scholar 

  23. Machiry, A., Tahiliani, R., Naik, M.: Dynodroid: an input generation system for android apps. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 224–234 (2013)

    Google Scholar 

  24. Mao, K., Harman, M., Jia, Y.: Sapienz: multi-objective automated testing for android applications. In: Proceedings of the 25th International Symposium on Software Testing and Analysis, pp. 94–105 (2016)

    Google Scholar 

  25. Margetis, G., Ntoa, S., Antona, M., Stephanidis, C.: Human-centered design of artificial intelligence. Handb. Hum. Factors Ergon., 1085–1106 (2021)

    Google Scholar 

  26. Mirzaei, N., Garcia, J., Bagheri, H., Sadeghi, A., Malek, S.: Reducing combinatorics in GUI testing of android applications. In: Proceedings of the 38th International Conference on Software Engineering, pp. 559–570 (2016)

    Google Scholar 

  27. Nakagawa, T., Munakata, K., Yamamoto, K.: Applying modified code entity-based regression test selection for manual end-to-end testing of commercial web applications. In: 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), pp. 1–6. IEEE (2019)

    Google Scholar 

  28. Palomba, F., Di Nucci, D., Panichella, A., Zaidman, A., De Lucia, A.: Lightweight detection of android-specific code smells: the adoctor project. In: IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE (2017)

    Google Scholar 

  29. Pan, M., Huang, A., Wang, G., Zhang, T., Li, X.: Reinforcement learning based curiosity-driven testing of android applications. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 153–164 (2020)

    Google Scholar 

  30. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

    Google Scholar 

  31. Shirazi, S., Alireza, N., Henze, A., Schmidt, R., Goldberg, B., Schmidt, H.: Insights into layout patterns of mobile user interfaces by an automatic analysis of android apps. In: Proceedings of the 5th ACM SIGCHI symposium on Engineering Interactive Computing Systems, pp. 275–284. Association for Computing Machinery (2013)

    Google Scholar 

  32. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  33. Soui, M., Chouchane, M., Gasmi, I., Mkaouer, M.W.: PLAIN: PLugin for predicting the usAbility of mobile user INterface. In: Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. SCITEPRESS - Science and Technology Publications (2017)

    Google Scholar 

  34. Soui, M., Chouchane, M., Mkaouer, M.W., Kessentini, M., Ghedira, K.: Assessing the quality of mobile graphical user interfaces using multi-objective optimization. Soft. Comput. 24(10), 7685–7714 (2020)

    Article  Google Scholar 

  35. Soui, M., Ghedira, K., Abed, M.: Evaluating user interface adaptation using the context of use. Int. J. Adapt. Resilient Auton. Syst. (IJARAS) 6(1), 1–24 (2015)

    Article  Google Scholar 

  36. Soui, M., Haddad, Z.: Deep learning-based model using densnet201 for mobile user interface evaluation. Int. J. Hum.-Comput. Interact. 1–14 (2023)

    Google Scholar 

  37. Su, Y., Liu, Z., Chen, C., Wang, J., Wang, Q.: OwlEyes-online: a fully automated platform for detecting and localizing UI display issues. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1500–1504 (2021)

    Google Scholar 

  38. Taylor, P.: Forecast number of mobile users worldwide 2020–2025 (2023). https://www.statista.com/statistics/218984/number-of-global-mobile-users-since-2010/

  39. Wei, L., Liu, Y., Cheung, S.C.: Taming android fragmentation: characterizing and detecting compatibility issues for android apps. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, pp. 226–237 (2016)

    Google Scholar 

  40. Xu, W.: Toward human-centered AI: a perspective from human-computer interaction. Interactions 26(4), 42–46 (2019)

    Article  Google Scholar 

  41. Yang, S., et al.: Static window transition graphs for android. Autom. Softw. Eng. 25, 833–873 (2018)

    Article  Google Scholar 

  42. Yang, W., Prasad, M.R., Xie, T.: A grey-box approach for automated GUI-model generation of mobile applications. In: Cortellessa, V., Varró, D. (eds.) FASE 2013. LNCS, vol. 7793, pp. 250–265. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37057-1_19

    Chapter  Google Scholar 

  43. Zeng, X., et al.: Automated test input generation for android: are we really there yet in an industrial case? In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 987–992 (2016)

    Google Scholar 

Download references

Acknowledgment

This paper is based upon work supported by The Academy of Scientific Technology (ASRT), Egypt. Scientists of Next Generation Scholarship Cycle (8).

Author information

Authors and Affiliations

  1. Center for Informatics Science, School of Information Technology and Computer Science (ITCS), Nile University, Giza, Egypt

    Arwa Ahmed, Ahmed Tamer Salah, Ghada Khoriba & Tamer Arafa

Authors
  1. Arwa Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmed Tamer Salah
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ghada Khoriba
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tamer Arafa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arwa Ahmed .

Editor information

Editors and Affiliations

  1. Bordeaux INP, Talence, France

    Mohamed Mosbah

  2. Dublin City University, Dublin, Ireland

    Tahar Kechadi

  3. ENSMA, Poitiers, France

    Ladjel Bellatreche

  4. University of Sfax, Sfax, Tunisia

    Faiez Gargouri

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ahmed, A., Salah, A.T., Khoriba, G., Arafa, T. (2024). Localizing Non-functional Code Bugs in User Interfaces Using Deep Learning Techniques. In: Mosbah, M., Kechadi, T., Bellatreche, L., Gargouri, F. (eds) Model and Data Engineering. MEDI 2023. Lecture Notes in Computer Science, vol 14396. Springer, Cham. https://doi.org/10.1007/978-3-031-49333-1_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-49333-1_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-49332-4

  • Online ISBN: 978-3-031-49333-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation