Skip to main content
SpringerLink
Log in

Banner layout retargeting with hierarchical reinforcement learning and variational autoencoder

  • 1168: Deep Pattern Discovery for Big Multimedia Data
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In many advertising areas, banners are often generated with different display sizes, so designers have to make huge efforts to retarget their designs to each size. Automating such retargeting process can greatly save time for designers and let them put creativity on new ads. This paper proposes a hierarchical reinforcement learning-based (HRL-based) method and a variational autoencoder-based (VAE-based) method by treating the automated banner retargeting problem as a layout retargeting task. The HRL and VAE models are trained separately to learn the scaling and positioning policy of the design elements from an original (base) layout. Hence, the proposed method can generate appropriate layouts for different target banner sizes. Meanwhile, evaluation metrics are proposed to assess the quality of generated layouts and are also reward conditions during the training process. To evaluate performances of the two models, SOTA methods such as Non-linear Inverse Optimization (NIO), Triangle Interpolation (TI), and Layout GAN (LGAN) are implemented and compared. Experimental results show that both HRL- and VAE-based methods retarget design layouts effectively, and the VAE model achieves better performance than the HRL model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Baluja S (2006) Browsing on small screens: recasting web-page segmentation into an efficient machine learning framework. In: Proceedings of the 15th international conference on World Wide Web. ACM, pp 33–42

  2. Cao Y, Lau R W H, Chan A B (2014) Look over here: attention-directing composition of manga elements. ACM Trans Graph 33(4):94:1–94:11

    Google Scholar 

  3. Damera-Venkata N, Bento J, O’Brien-Strain E (2011) Probabilistic document model for automated document composition. In: Proceedings of the 11th ACM symposium on Document engineering. ACM, pp 3–12

  4. Dietterich T G (2000) Hierarchical reinforcement learning with the maxq value function decomposition. J Artif Intell Res 13:227–303

    Article  MathSciNet  Google Scholar 

  5. Doersch C (2016) Tutorial on variational autoencoders. arXiv:1606.05908

  6. Earl D J, Deem M W (2005) Parallel tempering: theory, applications, and new perspectives. Phys Chem Chem Phys 7(23):3910–3916

    Article  Google Scholar 

  7. Hester T, Vecerik M, Pietquin, et al. (2018) Deep q-learning from demonstrations. In: Thirty-Second AAAI conference on artificial intelligence

  8. Hirashima Y (2009) A q-learning system for container marshalling with group-based learning model at container yard terminals. In: Proceedings of the international multiconference of engineers and computer scientists 2009 (IMECS 2009), vol 1

  9. Hu H, Zhang C, Liang Y (2021) A study on the automatic generation of banner layouts. Comput Electr Eng 93:107269

    Article  Google Scholar 

  10. Hua X-S (2018) Challenges and practices of large scale visual intelligence in the real-world. In: Proceedings of the 26th ACM international conference on multimedia, MM ’18. ACM, New York, pp 364–364

  11. Kingma D P, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  12. Kingma D P, Welling M (2013) Auto-encoding variational bayes. arXiv:1312.6114

  13. Kumar R, Talton J O, Ahmad S, Klemmer S R (2011) Bricolage: example-based retargeting for web design. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 2197–2206

  14. Li J, Yang J, Hertzmann A, Zhang J, Xu T (2019) Layoutgan: generating graphic layouts with wireframe discriminators. arXiv:1901.06767

  15. Li J, Yang J, Zhang J, Liu C, Wang C, Xu T (2020) Attribute-conditioned layout gan for automatic graphic design. IEEE Trans Vis Comput Graph

  16. Liu C K, Hertzmann A, Popović Z (2005) Learning physics-based motion style with nonlinear inverse optimization. ACM Trans Graph (TOG) 24 (3):1071–1081

    Article  Google Scholar 

  17. Liu L, Chen R, Wolf L, Cohen-Or D (2010) Optimizing photo composition. In: Computer graphics forum, vol 29. Wiley Online Library, pp 469–478

  18. Maheshwari P, Bansal N, Dwivedi S, Kumar R, Manerikar P, Srinivasan B V (2019) Exemplar based experience transfer. In: Proceedings of the 24th international conference on intelligent user interfaces. ACM, pp 673–680

  19. Mitchell T M (1999) Machine learning and data mining. Commun ACM 42(11):30–36

    Article  Google Scholar 

  20. Moon T K (1996) The expectation-maximization algorithm. IEEE Signal Process Mag 13(6):47–60

    Article  Google Scholar 

  21. Nowozin S, Lampert C H (2011) Structured learning and prediction in computer vision, vol 6 (3–4). Now Publishers Inc

  22. O’Donovan P (2015) Learning design: aesthetic models for color, layout, and typography. Ph.D. Thesis. University of Toronto (Canada)

  23. ODonovan P, Agarwala A, Hertzmann A (August 2014) Learning layouts for single-pagegraphic designs. IEEE Trans Vis Comput Graph 20(8):1200–1213

    Article  Google Scholar 

  24. O’Donovan P, Agarwala A, Hertzmann A (2015) Designscape: design with interactive layout suggestions. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ’15. ACM, New York, pp 1221–1224

  25. Osa T, Tangkaratt V, Sugiyama M (2019) Hierarchical reinforcement learning via advantage-weighted information maximization. arXiv:1901.01365

  26. Pang X, Cao Y, Lau R W H, Chan A B (2016) Directing user attention via visual flow on web designs. ACM Trans Graph 35(6):240:1–240:11

    Article  Google Scholar 

  27. Papernot N, McDaniel P, Goodfellow I, Jha S, Celik Z B, Swami A (2017) Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp 506–519

  28. Patil A G, Ben-Eliezer O, Perel O, Averbuch-Elor H (2019) Read: recursive autoencoders for document layout generation. arXiv:1909.00302

  29. Sandhaus P, Rabbath M, Boll S (2011) Employing aesthetic principles for automatic photo book layout. In: International conference on multimedia modeling. Springer, pp 84–95

  30. Sutton R S, Precup D, Singh S (1999) Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artif Intell 112(1-2):181–211

    Article  MathSciNet  Google Scholar 

  31. Tarkesh H, Atighehchian A, Nookabadi A S (2009) Facility layout design using virtual multi-agent system. J Intell Manuf 20(4):347

    Article  Google Scholar 

  32. Todi K, Weir D, Oulasvirta A (2016) Sketchplore: Sketch and explore with a layout optimiser. In: Proceedings of the 2016 ACM conference on designing interactive systems. ACM, pp 543–555

  33. Vempati S, Malayil K T, et al. (2019) Enabling hyper-personalisation: automated ad creative generation and ranking for fashion e-commerce. arXiv:1908.10139

  34. Watkins CJCH, Dayan P (1992) Q-learning. Mach Learn 8 (3–4):279–292

    MATH  Google Scholar 

  35. White CC (2001) Markov decision processes. Springer

  36. Yang X, Mei T, Xu Y-Q, Rui Y, Li S (2016) Automatic generation of visual-textual presentation layout. ACM Trans Multimed Comput Commun Applic (TOMM) 12(2):33

    Google Scholar 

  37. Zhang Y, Hu K, Ren P, Yang C, Xu W, Hua X-S (2017) Layout style modeling for automating banner design. In: Proceedings of the on thematic workshops of ACM multimedia 2017. ACM, pp 451–459

  38. Zheng X, Qiao X, Cao Y, Lau Rynson WH (2019) Content-aware generative modeling of graphic design layouts. ACM Trans Graph (TOG) 38(4):133

    Article  Google Scholar 

  39. Zhu C, Byrd R H, Lu P, Nocedal J (1997) Algorithm 778: L-bfgs-b: fortran subroutines for large-scale bound-constrained optimization. ACM Trans Math Softw (TOMS) 23(4):550–560

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Westlake University, Hangzhou, People’s Republic of China

    Hao Hu & Yanxue Liang

  2. Intelligent Industry Research Institute, Westlake University, Hangzhou, People’s Republic of China

    Chao Zhang

Authors
  1. Hao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanxue Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanxue Liang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Hao Hu and Chao Zhang contributed equally to this work.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, H., Zhang, C. & Liang, Y. Banner layout retargeting with hierarchical reinforcement learning and variational autoencoder. Multimed Tools Appl 81, 34417–34438 (2022). https://doi.org/10.1007/s11042-022-13325-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13325-w

Keywords

Search

Navigation