Bada Kim
ABSTRACT
The GUI building is an important part of web application development. Various studies such as WYSWYG web editor have been conducted to make this job convenient, where the job is composed of sketching of GUI and coding of HTML/CSS from the sketch. In this paper, we propose a novel way of web GUI building with computer vision and deep-learning. The proposed method requires only a hand-drawn sketch to build GUI. It recognizes web layout using computer vision algorithm, and web widgets using Faster R-CNN. With the recognized information, it makes HTML code automatically.
- S. P. Reiss, "Seeking the user interface," in Proc. 29th ACM/IEEE International Conference on Automated Software Engineering (ASE). ACM, Sep. 2014, pp. 103--114 Google Scholar
Digital Library
- Hung Viet Pham, Tam The Nguyen, Phong Minh Vu, Tung Thanh Nguyen, "Toward Mining Visual Log of Software", arXiv.org, 2016Google Scholar
- Tuan Anh Nguyen and Christoph Csallner, "Reverse Engineering Mobile Application User Interfaces with REMAUI", IEEE/ACM ASE, pp. 248--259, 2016.Google Scholar
- Girshick, R. B., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. CoRR, abs/1311.2524v5, 2014. Published in Proc. CVPR, 2014 Google ScholarDigital Library
- A. Salvador, X. Giró-i Nieto, F. Marqués, and S. Satoh, "Faster r-cnn features for instance search," in CVPR Workshops, 2016.Google Scholar
- S. Ren, K. He, R. Girshick, and J. Sun. "Faster R-CNN: Towards real-time object detection with region proposal networks". In NIPS, 2015 Google ScholarDigital Library
Index Terms
- Deep-learning based web UI automatic programming
Recommendations
Object detection for graphical user interface: old fashioned or deep learning or a combination?
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringDetecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element ...
Intelligent crawling of web applications for web archiving
WWW '12 Companion: Proceedings of the 21st International Conference on World Wide WebThe steady growth of the World Wide Web raises challenges regarding the preservation of meaningful Web data. Tools used currently by Web archivists blindly crawl and store Web pages found while crawling, disregarding the kind of Web site currently ...
Demonstrating intelligent crawling and archiving of web applications
CIKM '13: Proceedings of the 22nd ACM international conference on Information & Knowledge ManagementWe demonstrate here a new approach to Web archival crawling, based on an application-aware helper that drives crawls of Web applications according to their types (especially, according to their content management systems). By adapting the crawling ...
Comments