A study on the automatic generation of banner layouts

https://doi.org/10.1016/j.compeleceng.2021.107269Get rights and content

Highlights

  • Filling design elements to the layout to generate a banner

  • Aesthetic measures to evaluate the design layout

  • Traditional method to optimize the design layout based on designers’ experience

  • Optimize the design layout automatically with deep reinforcement learning

Abstract

This paper addresses the automatic banner design problem as a layout generation task, where a banner is generated by filling design elements to a given layout. We initially introduce the traditional method of generating design layouts used by designers. After that, we introduce a deep reinforcement learning (DRL) based method to learn the policy of generating design layouts. Evaluation metrics are introduced to assess the quality of design layouts generated by the two proposed methods. The experiment shows that, both methods can generate a layout for a banner of given size, and the DRL based method outperforms the traditional method by generating layouts of better quality. However, results based on the DRL method are affected by reward definitions and a long training process is needed to achieve a better performance.

Introduction

In modern life, banners appear in various medias such as advertisements, websites, and mobile phones since they play a key role in visual communication of various messages. Creating an original design is more artistic and some design principles should be complied to achieve pleasing visual appearance. Designing a beautiful and good quality banner is a difficult and time-consuming task, to refine it, since such tasks require professional skills to convey information clearly while also satisfying aesthetic goals. Moreover, many banners are created by non-experts, which consumes a lot of time and the quality is unsatisfying. As the need of creating new banners keeps growing rapidly, automatic tools for creating, adapting, and improving banners design could not only release the effort of designers, but also encourage new ideas and educate novice users.

A banner is often composed of an underlying theme, a background image, the brand logo in focus and affiliated text phrases [1]. These elements are usually taken from a library, placed according to a design layout and applied few image transformations to create a banner (Fig. 1). For designers, a layout is a blueprint gathering design elements to guide them to create design works. It determines the spatial arrangement of various design elements like text, logo, and others. It can be used to describe a group of banners, and to generate new banners with the same blueprint. Following design principles, it is composed of a set of rectangles, which determines the position and size of a design element. A high-quality layout can generate aesthetic pleasing banners and thus enhance information presentation, guide reader attention [2]. In recent years, design layout generation has received a growing interest in computer graphic community. Prior works focusing on layout generation are involved with design style, aesthetic measures, and perception [3,4].

Banner design is a complex problem involving layout, colors, fonts, design intent, and requirements gathering. In this work, however, we specifically focus on the layout generation. The layout generation is defined as specifying the locations and sizes of all elements in a banner. In this work, we assume that a set of texts and graphic elements are provided as inputs along with associated meta-data and their corresponding roles. The goal is to generate a visually satisfying banner by arranging these elements properly according to design principles. Although banner design involves a set of choices including colors, fonts and lines breaks, we leave selecting these variables for future work.

We present an automatic tool capable of creating and improving design layouts for design elements of arbitrary sizes. Due to the complexity of banner design, our approach has two stages. Given a set of design elements and their corresponding roles, we first initialize design layouts according to elements’ sizes. After that, we use energy terms to evaluate the quality of initial layouts and improve those with low quality. Since we do not consider elements’ style, color etc. in a design process, our quality evaluation metrics is different from those described in [6], which takes the darkness, brightness, HSV distribution of design elements into account. Finally, we introduce two methods for layout improvement. One is based on designers’ experience of adjusting bad designs. The other one is based on reinforcement learning, where agent learns the policy of layout optimization through interacting with environment. We believe this tool enables to assist designers to create new ideas and process large amount of designs in their daily work.

Our key technical contributions include:

  • We propose a traditional method that is used by designers to adjust design layouts according to design principles. Actions for adjusting positions of design elements are transformed into algorithms separately.

  • A deep reinforcement learning based method with two different reward functions for layout generation is introduced. Under our evaluation metrics, experiments are conducted to test the performance of different methods.

We test the proposed methods on real banner designs, producing competitive results with those created by professional designers. In the rest of this paper, we will first investigate prior work in Section 2 and clarify layout generation and evaluation in Section 3, then describe how to optimize design layouts in Section 4. Finally, we show an experimental study of proposed methods in Section 5 and conclude this work in Section 6.

Section snippets

Related works

In this section, approaches of automatic layout generation are discussed. They are classified into learning-based methods, constraint-based methods, and probabilistic graphical model-based methods.

A state-of-the-art learning approaches have been proposed to generate multi-size banners automatically by utilizing a triangle interpolation with learned style parameters and a multi-size style transfer technique [7]. LayoutGAN is used to synthesize layout by modeling geometric relations of different

Layout generation

This section describes our approach of generating layouts for a given set of design elements. The main idea is to initialize the position of bounding boxes of design elements of a layout and then optimize the layout with our approaches.

Layout optimization

In this section, we propose two methods to optimize the initial layouts based on evaluation metrics described above.

The first one is based on designer's experience of moving elements where algorithms are proposed for moving actions. The other one is based on reinforcement learning where DDPG [16] with different reward functions is introduced.

Training

We train the model on a dataset with 1800 layouts. Positions of bounding boxes within a layout are initialized randomly. The implementation details are as following. We simulate 500000 episodes. Each episode is limited to 500ms, which means there are at most 200 steps in a single episode. Initial state for each episode is sampled randomly in the state space. The learning rate is set to 0.00025 and the discounted factor is set to 0.99. The batch size is set to 128. The capacity of the transition

Conclusion

In this paper, we propose methods to automatically generate layouts of design banners. With the purpose of generating a design layout satisfying aesthetic measures, a layout is initialized by randomly positioning design elements, and then reorganize them with proposed methods. Proposed methods are tested and compared on a given banner dataset. Results of experiments show that the deep reinforcement learning based method outperforms the one of traditional method with respect to aesthetic

Author statement

Hao Hu: Conceptualization, Methodology, Software, Writing- Original draft preparation

Chao Zhang: Data curation, Validation, Visualization, Investigation.

Yanxue Liang: Writing- Reviewing and Editing, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Hao Hu received the PhD degree from Arts et Metiers ParisTech, France, in 2018. He is a senior researcher in Westlake University. His research interests include computer vision, perception, computer aided design and manufacturing.

References (24)

  • Hao Hu et al.

    Over-constraints detection and resolution in geometric equation systems

    Comput-Aided Des

    (2017)
  • Vempati, Sreekanth, and Korah T. Malayil. “Enabling hyper-personalization: Automated ad creative generation and ranking...
  • Y. Cao et al.

    Look over here: Attention-directing composition of manga elements

    ACM Trans Graph

    (July 2014)
  • N. Damera-Venkata et al.

    Probabilistic document model for automated document composition

  • X. Yang et al.

    Automatic generation of visual-textual presentation layout

    ACM Trans Multimedia Comput, Commun Appl (TOMM)

    (2016)
  • P. O'Donovan

    Learning Design: Aesthetic Models for Color, Layout, and Typography

    (2015)
  • Y. Wu et al.

    Monet: A system for reliving your memories by theme-based photo storytelling

    IEEE Trans Multimedia

    (2016)
  • Y. Zhang et al.

    Layout style modeling for automating banner design

  • J. Li, J. Yang, A. Hertzmann, J. Zhang, and T. Xu, “Layoutgan: Generating graphic layouts with wireframe...
  • H. Tarkesh et al.

    Facility layout design using virtual multi-agent system

    J Intell Manuf

    (2009)
  • Y. Hirashima

    A q-learning system for container marshalling with group-based learning model at container yard terminals

  • P. ODonovan et al.

    Learning layouts for single-page graphic designs

    IEEE Trans Vis Comput Graph

    (2014)
  • Cited by (0)

    Hao Hu received the PhD degree from Arts et Metiers ParisTech, France, in 2018. He is a senior researcher in Westlake University. His research interests include computer vision, perception, computer aided design and manufacturing.

    Chao Zhang received his master's degree from Chang'An University in 2019. He is a computer vision engineer in Intelligent Industry Research Institute of Westlake University. His research interests include Deep Learning and Reinforcement Learning applications.

    Yanxue Liang Received his master's degree from Shanghai JiaoTong University in 2003 and PhD degree from Tokyo Institute of Technology in 2008. He is now a professor in Westlake University. His current research interests include perception of robotics, CAD, and CAM.

    This paper is for special section VSI-hci. Reviews processed and recommended for publication by Guest Editor Dr. Shengzong Zhou.

    This paper was recommended for publication by Associate Editor Dr. M. Malek.

    1

    Equal contribution

    View full text