Skip to main content
SpringerLink
Log in

Code generation from a graphical user interface via attention-based encoder–decoder model

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Code generation from graphical user interface images is a promising area of research. Recent progress on machine learning methods made it possible to transform user interface into the code using several methods. The encoder–decoder framework represents one of the possible ways to tackle code generation tasks. Our model implements the encoder–decoder framework with an attention mechanism that helps the decoder to focus on a subset of salient image features when needed. Our attention mechanism also helps the decoder to generate token sequences with higher accuracy. Experimental results show that our model outperforms previously proposed models on the pix2code benchmark dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Girshick, R.B.: Fast R-CNN. CoRR arXiv:1504.08083 (2015)

  2. Beltramelli, T.: pix2code: Generating code from a graphical user interface screenshot. CoRR arxiv:1705.07962 (2017)

  3. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. CoRR arXiv:1502.03044 (2015)

  4. Liu, Y., Hu, Q., Shu, K.: Improving pix2code based bi-directional lstm. 2018 IEEE international conference on automation, electronics and electrical engineering (AUTEEE) p. 220–223 (2018)

  5. Zhu, Z., Xue, Z., Yuan, Z.: Automatic graphics program generation using attention-based hierarchical decoder. CoRR arXiv:1810.11536 (2018)

  6. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate (2014)

  7. Tang, H.L., Chien, S.C., Cheng, W.H., Chen, Y.Y., Hua, K.L.: Multi-cue pedestrian detection from 3d point cloud data. In: 2017 IEEE international conference on multimedia and expo (ICME), pp. 1279–1284. IEEE (2017)

  8. Hua, K.L., Hidayati, S.C., He, F.L., Wei, C.P., Wang, Y.C.F.: Context-aware joint dictionary learning for color image demosaicking. J. Vis. Commun. Image Represent. 38, 230–245 (2016)

    Article  Google Scholar 

  9. Tan, D.S., Chen, W.Y., Hua, K.L.: Deepdemosaicking: adaptive image demosaicking via multiple deep fully convolutional networks. IEEE Trans. Image Process. 27(5), 2408–2419 (2018)

    Article  MathSciNet  Google Scholar 

  10. Sanchez-Riera, J., Hua, K.L., Hsiao, Y.S., Lim, T., Hidayati, S.C., Cheng, W.H.: A comparative study of data fusion for rgb-d based visual recognition. Pattern Recogn. Lett. 73, 1–6 (2016)

    Article  Google Scholar 

  11. Hidayati, S.C., Hua, K.L., Cheng, W.H., Sun, S.W.: What are the fashion trends in new york? In: Proceedings of the 22nd ACM international conference on multimedia, pp. 197–200 (2014)

  12. Sharma, V., Srinivasan, K., Chao, H.C., Hua, K.L., Cheng, W.H.: Intelligent deployment of uavs in 5g heterogeneous communication environment for improved coverage. J. Netw. Comput. Appl. 85, 94–105 (2017)

    Article  Google Scholar 

  13. Chen, X., Zitnick, C.L.: Learning a recurrent visual representation for image caption generation. CoRR arXiv:1411.5654 (2014)

  14. Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., Yuille, A.: Deep captioning with multimodal recurrent neural networks (m-rnn) (2015)

  15. Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Chua, T.: SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning. CoRR arxiv:1611.05594 (2016)

  16. Lu, J., Xiong, C., Parikh, D., Socher, R.: Knowing when to look: adaptive attention via A visual sentinel for image captioning. CoRR arXiv:1612.01887 (2016)

  17. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  18. Ba, L.J., Kiros, J.R., Hinton, G.E.: Layer normalization. CoRR arxiv:1607.06450 (2016)

  19. Cho, K., van Merrienboer, B., Gülçehre, Ç., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR arXiv:1406.1078 (2014)

  20. Luong, M., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. CoRR arXiv:1508.04025 (2015)

  21. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)

  22. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M.S., Berg, A.C., Li, F.: Imagenet large scale visual recognition challenge. CoRR arXiv:1409.0575 (2014)

Download references

Author information

Authors and Affiliations

  1. Department of Arts and Design, National Taipei University of Education, Taipei, Taiwan

    Wen-Yin Chen

  2. Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan

    Pavol Podstreleny & Kai-Lung Hua

  3. Institute of Electronics, National Chiao Tung University, Hsinchu, Taiwan

    Wen-Huang Cheng

  4. Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan

    Yung-Yao Chen

Authors
  1. Wen-Yin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pavol Podstreleny
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wen-Huang Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yung-Yao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kai-Lung Hua
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kai-Lung Hua.

Additional information

Communicated by B.-K. Bao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, WY., Podstreleny, P., Cheng, WH. et al. Code generation from a graphical user interface via attention-based encoder–decoder model. Multimedia Systems 28, 121–130 (2022). https://doi.org/10.1007/s00530-021-00804-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-021-00804-7

Keywords

Search

Navigation