skip to main content
10.1145/3472301.3484340acmotherconferencesArticle/Chapter ViewAbstractPublication PagesihcConference Proceedingsconference-collections
research-article

Using Deep Learning to Support the User Interface Design of Mobile Applications with App Inventor

Authors Info & Claims
Published:18 October 2021Publication History

ABSTRACT

Transforming a user interface sketch into a wireframe using App Inventor has been a common need both in computing education and the development of applications by end users. As this part of mobile application development is challenging and time-consuming, we present an approach that automates this process through the prototyping of user interfaces. The Sketch2aia approach employs deep learning to detect the user interface components and their position in a hand-drawn sketch, creating an intermediate representation of the user interface and automatically generating the App Inventor code for the respective wireframe. The approach achieves an average classification accuracy of the components of the user interface of 87.72% and results of a preliminary evaluation with users indicate that it generates wireframes that resemble sketches in terms of visual similarity. The approach is available as a web tool and can be used to support the teaching of user interface design effectively and efficiently, as well as the development of mobile applications by end users.

References

  1. Nathalia d. C. Alves, Christiane Gresse von Wangenheim, and Jean C. R. Hauck, 2020. A Large-scale Analysis of App Inventor Projects. arXiv:2006.11327 [cs.CY].Google ScholarGoogle Scholar
  2. Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann, 2019. Software engineering for machine learning: a case study. In Proc. of the 41st Int. Conference on Software Engineering: Software Engineering in Practice, IEEE Press, 291--300.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Batuhan Aşıroğlu, Büşta R. Mete, Eyyüp Yıldız, Yağız Nalçakan, Alper Sezen, Mustafa Dağtekin, and Tolga Ensari, 2019. Automatic HTML Code Generation from Mock-up Images Using Machine Learning Techniques. In Proc. of the Scientific Meeting on Electrical-Electronics and Biomedical Engineering and Computer Science, Istanbul, Turkey, 1--4.Google ScholarGoogle ScholarCross RefCross Ref
  4. Barbara R. Barricelli, Fabio Cassano, Daniela Fogli, and Antonio Piccinno, 2019. End-user development, end-user programming and end-user software engineering: A systematic mapping study. Journal of Systems and Software, 149, 101--137.Google ScholarGoogle ScholarCross RefCross Ref
  5. Victor R. Basili. 1992. Software Modeling and Measurement: The Goal/Question/Metric Paradigm. University of Maryland, CS-TR-2956, UMIACS-TR-92-96, College Park, MD, USA.Google ScholarGoogle Scholar
  6. Daniel S. Baulé, Christiane Gresse von Wangenheim, Aldo von Wangenheim and Jean C. R. Hauck, 2020. Recent Progress in Automated Code Generation from GUI Images using Machine Learning Techniques. JOURNAL OF UNIVERSAL COMPUTER SCIENCE, v. 26, p. 1095--1127.Google ScholarGoogle ScholarCross RefCross Ref
  7. Tony Beltramelli. 2019. Hack your design sprint: wireframes to prototype in under 5 minutes"; Medium, https://uxdesign.cc/hack-you-design-sprints-wireframes-to-prototype-in-under-5-minutes-b7b95c8b2aa2Google ScholarGoogle Scholar
  8. Tony Beltramelli. 2018. pix2code: Generating Code from a Graphical User Interface Screenshot. In Proc. of the SIGCHI Symposium on Engineering Interactive Computing Systems, ACM, New York, NY, USA, Article 3, 1--6.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Alexey Bochkovskiy. 2019. Yolo-v3 and Yolo-v2 for Windows and Linux, Github, https://github.com/AlexeyAB/darknet/Google ScholarGoogle Scholar
  10. Tim Brown. 2008. Design thinking. Harvard Business Review, 86(5), 84--92.Google ScholarGoogle Scholar
  11. John M. Carroll, 2013. Human Computer Interaction - brief intro. In: The Encyclopedia of Human-Computer Interaction, 2nd Ed.. Aarhus, Denmark: The Interaction Design Foundation.Google ScholarGoogle Scholar
  12. Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu, 2018. From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In Proc. of the Int. Conference on Software Engineering, ACM, New York, NY, USA, 665--676.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. CSTA. 2017. K-12 Computer Science Framework, http://k12cs.org/Google ScholarGoogle Scholar
  14. Miriam N. F. Ferreira, Christiane Gresse von Wangenheim, Raul M. Filho, Fernando d. C. Pinheiro, and Jean C. R. Hauck, 2019a. Learning user interface design and the development of mobile applications in middle school. ACM Interactions, 26(4).Google ScholarGoogle Scholar
  15. Miriam N. F. Ferreira, Fernando Pinheiro, Raul M. Filho, and Christiane Gresse von Wangenheim, 2019b. Ensinando Design de Interface de Usuário na Educação Básica: Um Mapeamento Sistemático do Estado da Arte e Prática. In Anais do Workshop de Informática na Escola, Brasília, Brasil.Google ScholarGoogle Scholar
  16. Xiaofei Ge. 2019. Android GUI search using hand-drawn sketches. In Proc. of the 41st Int. Conference on Software Engineering, IEEE Press, 141--143.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Yi Han, Jun He, and Qiwen Dong, 2018. CSSSketch2Code: An Automatic Method to Generate Web Pages with CSS Style. In Proc. of the 2nd Int. Conference on Advances in Artificial Intelligence, Barcelona, Spain, 29--35.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Rex Hartson and Pardha Pyla, 2012. The UX Book: Process and Guidelines for Ensuring a Quality User Experience, Morgan Kaufmann.Google ScholarGoogle Scholar
  19. Forrest Huang, John F. Canny, and Jeffrey Nichols, 2019. Swire: Sketch-based User Interface Retrieval. In Proc. of the CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 1--10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Vanita Jain, Piyush Agrawal, Subham Banga, Rishabh Kapoor and Shashwat Gulyani, 2019. Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Network. arXiv:1910.08930 [cs.CV].Google ScholarGoogle Scholar
  21. Ashwin Kumar, 2018. Automated front-end development using deep learning, Medium Insight.Google ScholarGoogle Scholar
  22. Craig Larman, Victor R. Basili, 2003. Iterative and incremental development, IEEE Computer, 36(6), 47--56.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Yanbin Liu, Qidi Hu, and Kunxian Shu, 2018. Improving pix2code based Bidirectional LSTM. In Proc. of the IEEE Int. Conference on Automation, Electronics and Electrical Engineering, Shenyang, China, 220--223.Google ScholarGoogle Scholar
  24. MIT. 2020. App Inventor, http://http://appinventor.mit.eduGoogle ScholarGoogle Scholar
  25. Thomas M. Mitchell. 1997. Machine Learning. McGraw-Hill Education, New York.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Kevin Moran, Michael Curcio, and Denys Poshyvanyk, 2018. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. IEEE Transactions on Software Engineering, 46(2), 196--221.Google ScholarGoogle ScholarCross RefCross Ref
  27. Eni Mustafaraj, Franklyn Turbak, and Maja Svanberg, 2017. Identifying original projects in App Inventor. In Proc. of the 30th Int. Flairs Conference, Marco Island, USA.Google ScholarGoogle Scholar
  28. Ipek Ozkaya. 2019. Are DevOps and Automation Our Next Silver Bullet? IEEE Software, 36(4), 3--95.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Vinoth S. Pandian, Sarah Suleri, and Matthais Jarke, 2020. Blu: What GUIs are made of. In Proc. of the 25th Int. Conference on Intelligent User Interfaces, ACM, New York, NY, USA, 81--82.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Fabio Paternò. 2013. End User Development: Survey of an Emerging Field for Empowering People. ISRN Software Engineering, vol. 2013, Article ID 532659.Google ScholarGoogle ScholarCross RefCross Ref
  31. Fernando d. C. Pinheiro, Christiane Gresse von Wangenheim, and Raul M. Filho, 2018. Teaching Software Engineering in K-12 Education: A Systematic Mapping Study. Informatics in Education, 17(2), 167--206.Google ScholarGoogle ScholarCross RefCross Ref
  32. Joseph Redmon. 2016; Darknet: Open Source Neural Networks in C. http://pjreddie.com/darknet/.Google ScholarGoogle Scholar
  33. Joseph Redmon and Ali Farhadi, 2019. YOLOv3: An Incremental Improvement. arXiv preprint arXiv:1804.02767.Google ScholarGoogle Scholar
  34. Brian D. Ripley. 1996. Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Alex Robinson. 2018. Sketch2code: Generating a website from a paper mockup. Dissertation, University of Bristol, UK.Google ScholarGoogle Scholar
  36. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei, 2014. ImageNet Large Scale Visual Recognition Challenge. arXiv preprint arXiv:1409.0575.Google ScholarGoogle Scholar
  37. Tiago S. d. Silva, Angela Martin, Frank Maurer, and Milene Silveira, 2011. User-Centered Design and Agile Methods: A Systematic Review. In Proc. of the Agile Conference, Salt Lake City, UT, USA, 77--86.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Sarah Suleri, Vinoth S. Pandian, Svetlana Shishkovets, and Matthais Jarke, 2019. Eve: A Sketch-based Software Prototyping Workbench. In Proc. of the Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 1--6.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Mike Tissenbaum, Josh Sheldon, and Hal Abelson, 2019. From computational thinking to computational action. Communications of the ACM, 62(3), 34--36.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Emil Wallner. 2018. Turning Design Mockups into Code with Deep Learning. Floydhub (2018) Article available at: https://blog.floydhub.com/turning-design-mockups-into-code-with-deep-learningGoogle ScholarGoogle Scholar
  41. Anthony I. Wasserman. 2010. Software engineering issues for mobile application development. In Proc. of the Workshop on Future of Software Engineering Research. ACM, New York, NY, USA, 397--400.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Kentaro Wada. 2016. labelme: Image Polygonal Annotation with Python, Github, https://github. com/wkentaro/labelmeGoogle ScholarGoogle Scholar
  43. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson, 2014. How transferable are features in deep neural networks?. In Proc. of the 27th Int. Conference on Neural Information Processing Systems, Montreal, Canada, 3320--3328.Google ScholarGoogle Scholar
  44. Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye, 2019. Object Detection in 20 Years: A Survey, arXiv:1905.05055v2 [cs.CV].Google ScholarGoogle Scholar

Index Terms

  1. Using Deep Learning to Support the User Interface Design of Mobile Applications with App Inventor

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in
      • Published in

        cover image ACM Other conferences
        IHC '21: Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems
        October 2021
        523 pages
        ISBN:9781450386173
        DOI:10.1145/3472301

        Copyright © 2021 ACM

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 18 October 2021

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed limited

        Acceptance Rates

        IHC '21 Paper Acceptance Rate29of77submissions,38%Overall Acceptance Rate331of973submissions,34%
      • Article Metrics

        • Downloads (Last 12 months)49
        • Downloads (Last 6 weeks)5

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader