ABSTRACT
Transforming a user interface sketch into a wireframe using App Inventor has been a common need both in computing education and the development of applications by end users. As this part of mobile application development is challenging and time-consuming, we present an approach that automates this process through the prototyping of user interfaces. The Sketch2aia approach employs deep learning to detect the user interface components and their position in a hand-drawn sketch, creating an intermediate representation of the user interface and automatically generating the App Inventor code for the respective wireframe. The approach achieves an average classification accuracy of the components of the user interface of 87.72% and results of a preliminary evaluation with users indicate that it generates wireframes that resemble sketches in terms of visual similarity. The approach is available as a web tool and can be used to support the teaching of user interface design effectively and efficiently, as well as the development of mobile applications by end users.
- Nathalia d. C. Alves, Christiane Gresse von Wangenheim, and Jean C. R. Hauck, 2020. A Large-scale Analysis of App Inventor Projects. arXiv:2006.11327 [cs.CY].Google Scholar
- Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann, 2019. Software engineering for machine learning: a case study. In Proc. of the 41st Int. Conference on Software Engineering: Software Engineering in Practice, IEEE Press, 291--300.Google ScholarDigital Library
- Batuhan Aşıroğlu, Büşta R. Mete, Eyyüp Yıldız, Yağız Nalçakan, Alper Sezen, Mustafa Dağtekin, and Tolga Ensari, 2019. Automatic HTML Code Generation from Mock-up Images Using Machine Learning Techniques. In Proc. of the Scientific Meeting on Electrical-Electronics and Biomedical Engineering and Computer Science, Istanbul, Turkey, 1--4.Google ScholarCross Ref
- Barbara R. Barricelli, Fabio Cassano, Daniela Fogli, and Antonio Piccinno, 2019. End-user development, end-user programming and end-user software engineering: A systematic mapping study. Journal of Systems and Software, 149, 101--137.Google ScholarCross Ref
- Victor R. Basili. 1992. Software Modeling and Measurement: The Goal/Question/Metric Paradigm. University of Maryland, CS-TR-2956, UMIACS-TR-92-96, College Park, MD, USA.Google Scholar
- Daniel S. Baulé, Christiane Gresse von Wangenheim, Aldo von Wangenheim and Jean C. R. Hauck, 2020. Recent Progress in Automated Code Generation from GUI Images using Machine Learning Techniques. JOURNAL OF UNIVERSAL COMPUTER SCIENCE, v. 26, p. 1095--1127.Google ScholarCross Ref
- Tony Beltramelli. 2019. Hack your design sprint: wireframes to prototype in under 5 minutes"; Medium, https://uxdesign.cc/hack-you-design-sprints-wireframes-to-prototype-in-under-5-minutes-b7b95c8b2aa2Google Scholar
- Tony Beltramelli. 2018. pix2code: Generating Code from a Graphical User Interface Screenshot. In Proc. of the SIGCHI Symposium on Engineering Interactive Computing Systems, ACM, New York, NY, USA, Article 3, 1--6.Google ScholarDigital Library
- Alexey Bochkovskiy. 2019. Yolo-v3 and Yolo-v2 for Windows and Linux, Github, https://github.com/AlexeyAB/darknet/Google Scholar
- Tim Brown. 2008. Design thinking. Harvard Business Review, 86(5), 84--92.Google Scholar
- John M. Carroll, 2013. Human Computer Interaction - brief intro. In: The Encyclopedia of Human-Computer Interaction, 2nd Ed.. Aarhus, Denmark: The Interaction Design Foundation.Google Scholar
- Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu, 2018. From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In Proc. of the Int. Conference on Software Engineering, ACM, New York, NY, USA, 665--676.Google ScholarDigital Library
- CSTA. 2017. K-12 Computer Science Framework, http://k12cs.org/Google Scholar
- Miriam N. F. Ferreira, Christiane Gresse von Wangenheim, Raul M. Filho, Fernando d. C. Pinheiro, and Jean C. R. Hauck, 2019a. Learning user interface design and the development of mobile applications in middle school. ACM Interactions, 26(4).Google Scholar
- Miriam N. F. Ferreira, Fernando Pinheiro, Raul M. Filho, and Christiane Gresse von Wangenheim, 2019b. Ensinando Design de Interface de Usuário na Educação Básica: Um Mapeamento Sistemático do Estado da Arte e Prática. In Anais do Workshop de Informática na Escola, Brasília, Brasil.Google Scholar
- Xiaofei Ge. 2019. Android GUI search using hand-drawn sketches. In Proc. of the 41st Int. Conference on Software Engineering, IEEE Press, 141--143.Google ScholarDigital Library
- Yi Han, Jun He, and Qiwen Dong, 2018. CSSSketch2Code: An Automatic Method to Generate Web Pages with CSS Style. In Proc. of the 2nd Int. Conference on Advances in Artificial Intelligence, Barcelona, Spain, 29--35.Google ScholarDigital Library
- Rex Hartson and Pardha Pyla, 2012. The UX Book: Process and Guidelines for Ensuring a Quality User Experience, Morgan Kaufmann.Google Scholar
- Forrest Huang, John F. Canny, and Jeffrey Nichols, 2019. Swire: Sketch-based User Interface Retrieval. In Proc. of the CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 1--10.Google ScholarDigital Library
- Vanita Jain, Piyush Agrawal, Subham Banga, Rishabh Kapoor and Shashwat Gulyani, 2019. Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Network. arXiv:1910.08930 [cs.CV].Google Scholar
- Ashwin Kumar, 2018. Automated front-end development using deep learning, Medium Insight.Google Scholar
- Craig Larman, Victor R. Basili, 2003. Iterative and incremental development, IEEE Computer, 36(6), 47--56.Google ScholarDigital Library
- Yanbin Liu, Qidi Hu, and Kunxian Shu, 2018. Improving pix2code based Bidirectional LSTM. In Proc. of the IEEE Int. Conference on Automation, Electronics and Electrical Engineering, Shenyang, China, 220--223.Google Scholar
- MIT. 2020. App Inventor, http://http://appinventor.mit.eduGoogle Scholar
- Thomas M. Mitchell. 1997. Machine Learning. McGraw-Hill Education, New York.Google ScholarDigital Library
- Kevin Moran, Michael Curcio, and Denys Poshyvanyk, 2018. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. IEEE Transactions on Software Engineering, 46(2), 196--221.Google ScholarCross Ref
- Eni Mustafaraj, Franklyn Turbak, and Maja Svanberg, 2017. Identifying original projects in App Inventor. In Proc. of the 30th Int. Flairs Conference, Marco Island, USA.Google Scholar
- Ipek Ozkaya. 2019. Are DevOps and Automation Our Next Silver Bullet? IEEE Software, 36(4), 3--95.Google ScholarDigital Library
- Vinoth S. Pandian, Sarah Suleri, and Matthais Jarke, 2020. Blu: What GUIs are made of. In Proc. of the 25th Int. Conference on Intelligent User Interfaces, ACM, New York, NY, USA, 81--82.Google ScholarDigital Library
- Fabio Paternò. 2013. End User Development: Survey of an Emerging Field for Empowering People. ISRN Software Engineering, vol. 2013, Article ID 532659.Google ScholarCross Ref
- Fernando d. C. Pinheiro, Christiane Gresse von Wangenheim, and Raul M. Filho, 2018. Teaching Software Engineering in K-12 Education: A Systematic Mapping Study. Informatics in Education, 17(2), 167--206.Google ScholarCross Ref
- Joseph Redmon. 2016; Darknet: Open Source Neural Networks in C. http://pjreddie.com/darknet/.Google Scholar
- Joseph Redmon and Ali Farhadi, 2019. YOLOv3: An Incremental Improvement. arXiv preprint arXiv:1804.02767.Google Scholar
- Brian D. Ripley. 1996. Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge.Google ScholarDigital Library
- Alex Robinson. 2018. Sketch2code: Generating a website from a paper mockup. Dissertation, University of Bristol, UK.Google Scholar
- Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei, 2014. ImageNet Large Scale Visual Recognition Challenge. arXiv preprint arXiv:1409.0575.Google Scholar
- Tiago S. d. Silva, Angela Martin, Frank Maurer, and Milene Silveira, 2011. User-Centered Design and Agile Methods: A Systematic Review. In Proc. of the Agile Conference, Salt Lake City, UT, USA, 77--86.Google ScholarDigital Library
- Sarah Suleri, Vinoth S. Pandian, Svetlana Shishkovets, and Matthais Jarke, 2019. Eve: A Sketch-based Software Prototyping Workbench. In Proc. of the Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 1--6.Google ScholarDigital Library
- Mike Tissenbaum, Josh Sheldon, and Hal Abelson, 2019. From computational thinking to computational action. Communications of the ACM, 62(3), 34--36.Google ScholarDigital Library
- Emil Wallner. 2018. Turning Design Mockups into Code with Deep Learning. Floydhub (2018) Article available at: https://blog.floydhub.com/turning-design-mockups-into-code-with-deep-learningGoogle Scholar
- Anthony I. Wasserman. 2010. Software engineering issues for mobile application development. In Proc. of the Workshop on Future of Software Engineering Research. ACM, New York, NY, USA, 397--400.Google ScholarDigital Library
- Kentaro Wada. 2016. labelme: Image Polygonal Annotation with Python, Github, https://github. com/wkentaro/labelmeGoogle Scholar
- Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson, 2014. How transferable are features in deep neural networks?. In Proc. of the 27th Int. Conference on Neural Information Processing Systems, Montreal, Canada, 3320--3328.Google Scholar
- Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye, 2019. Object Detection in 20 Years: A Survey, arXiv:1905.05055v2 [cs.CV].Google Scholar
Index Terms
Using Deep Learning to Support the User Interface Design of Mobile Applications with App Inventor
Recommendations
Analyzing structural similarity of user interface layouts of Android apps using deep learning
IHC '22: Proceedings of the 21st Brazilian Symposium on Human Factors in Computing SystemsThe layout of graphical user interfaces (GUIs) takes into consideration the distribution of buttons, images, texts, and other components that are important factors for a positive experience of interactivity and usability. Therefore, it may be helpful to ...
The University of Alberta user interface management system
In this paper the design and implementation of the University of Alberta user interface management system (UIMS) is discussed. This UIMS is based on the Seeheim model of user interfaces, which divides the user interface into three separate components. ...
Comments