research-article

Two-stage Content-Aware Layout Generation for Poster Designs

Authors:

Liansheng Zhuang,

Zihan ZhouAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 8415 - 8423

https://doi.org/10.1145/3581783.3612275

Published: 27 October 2023 Publication History

Abstract

Automatic layout generation models can generate numerous design layouts in a few seconds, which significantly reduces the amount of repetitive work for designers. However, most of these models consider the layout generation task as arranging layout elements with different attributes on a blank canvas, thus struggle to handle the case when an image is used as the layout background. Additionally, existing layout generation models often fail to incorporate explicit aesthetic principles such as alignment and non-overlap, and neglect implicit aesthetic principles which are hard to model. To address these issues, this paper proposes a two-stage content-aware layout generation framework for poster layout generation. Our framework consists of an aesthetics-conditioned layout generation module and a layout ranking module. The diffusion model based layout generation module utilizes an aesthetics-guided layout denoising process to sample layout proposals that meet explicit aesthetic constraints. The Auto-Encoder based layout ranking module then measures the distance between those proposals and real designs to determine the layout that best meets implicit aesthetic principles. Quantitative and qualitative experiments demonstrate that our method outperforms state-of-the-art content-aware layout generation models.

References

[1]

Diego Martín Arroyo, Janis Postels, and Federico Tombari. 2021. Variational Transformer Networks for Layout Generation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 13637--13647. https://doi.org/10.1109/CVPR46437.2021.01343

[2]

Yunning Cao, Ye Ma, Min Zhou, Chuanbin Liu, Hongtao Xie, Tiezheng Ge, and Yuning Jiang. 2022. Geometry Aligned Variational Transformer for Image-Conditioned Layout Generation. In Proceedings of the 30th ACM International Conference on Multimedia (Lisboa, Portugal) (MM '22). Association for Computing Machinery, New York, NY, USA, 1561--1571. https://doi.org/10.1145/3503161.3548332

Digital Library

[3]

Shang Chai, Liansheng Zhuang, and Fengying Yan. 2023. LayoutDM: Transformer-based Diffusion Model for Layout Generation. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). arXiv eprint. https://doi.org/10.48550/arXiv.2305.02567

[4]

Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil Bharath. 2017. Generative Adversarial Networks: An Overview. IEEE Signal Processing Magazine, Vol. 35 (10 2017). https://doi.org/10.1109/MSP.2017.2765202

[5]

Taylor Denouden, Rick Salay, Krzysztof Czarnecki, Vahdat Abdelzad, Buu Phan, and Sachin Vernekar. 2018. Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance. arXiv preprint arXiv:1812.02765 (2018).

[6]

Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion Models Beat GANs on Image Synthesis. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 8780--8794. https://proceedings.neurips.cc/paper/2021/file/49ad23d1ec9fa4bd8d77d02681df5cfa-Paper.pdf

[7]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.

[8]

Shunan Guo, Zhuochen Jin, Fuling Sun, Jingwen Li, Zhaorui Li, Yang Shi, and Nan Cao. 2021. Vinci: An Intelligent Graphic Design System for Generating Advertising Posters. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 577, 17 pages. https://doi.org/10.1145/3411764.3445117

Digital Library

[9]

Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry Davis, Vijay Mahadevan, and Abhinav Shrivastava. 2021. LayoutTransformer: Layout Generation and Completion with Self-attention. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 984--994. https://doi.org/10.1109/ICCV48922.2021.00104

[10]

Jonathan Ho. 2022. Classifier-Free Diffusion Guidance. ArXiv, Vol. abs/2207.12598 (2022).

[11]

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020a. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 6840--6851. https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf

[12]

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020b. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, Vol. 33 (2020), 6840--6851.

[13]

Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. 2022. ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech. In Proceedings of the 30th ACM International Conference on Multimedia (Lisboa, Portugal) (MM '22). Association for Computing Machinery, New York, NY, USA, 2595--2605. https://doi.org/10.1145/3503161.3547855

Digital Library

[14]

Ali Jahanian, Jerry Liu, Qian Lin, Daniel Tretter, Eamonn O'Brien-Strain, Seungyon Claire Lee, Nic Lyons, and Jan Allebach. 2013. Recommendation System for Automatic Design of Magazine Covers. In Proceedings of the 2013 International Conference on Intelligent User Interfaces (Santa Monica, California, USA) (IUI '13). Association for Computing Machinery, New York, NY, USA, 95--106. https://doi.org/10.1145/2449396.2449411

Digital Library

[15]

Zhaoyun Jiang, Shizhao Sun, Jihua Zhu, Jian-Guang Lou, and Dongmei Zhang. 2022. Coarse-to-Fine Generative Modeling for Graphic Layouts. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36 (06 2022), 1096--1103. https://doi.org/10.1609/aaai.v36i1.19994

[16]

Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. 2019. LayoutVAE: Stochastic Scene Layout Generation From a Label Set. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 9894--9903. https://doi.org/10.1109/ICCV.2019.00999

[17]

Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. 2021a. Constrained Graphic Layout Generation via Latent Optimization. In ACM International Conference on Multimedia (MM '21). 88--96. https://doi.org/10.1145/3474085.3475497

Digital Library

[18]

Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. 2021b. Constrained Graphic Layout Generation via Latent Optimization. In Proceedings of the 29th ACM International Conference on Multimedia (Virtual Event, China) (MM '21). Association for Computing Machinery, New York, NY, USA, 88--96. https://doi.org/10.1145/3474085.3475497

Digital Library

[19]

Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. CoRR, Vol. abs/1312.6114 (2014).

[20]

Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa. 2022. BLT: bidirectional layout transformer for controllable layout generation. In European Conference on Computer Vision. Springer, 474--490.

Digital Library

[21]

Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B. Le, Haifeng Gong, Ming-Hsuan Yang, and Weilong Yang. 2020. Neural Design Network: Graphic Layout Generation with Constraints. In Computer Vision - ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 491--506.

Digital Library

[22]

Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. 2021. LayoutGAN: Synthesizing Graphic Layouts With Vector-Wireframe Adversarial Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, 7 (2021), 2388--2399. https://doi.org/10.1109/TPAMI.2019.2963663

[23]

Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. 2022. Diffusion-LM Improves Controllable Text Generation. ArXiv, Vol. abs/2205.14217 (2022).

[24]

Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. DesignScape: Design with Interactive Layout Suggestions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 1221--1224. https://doi.org/10.1145/2702123.2702149

Digital Library

[25]

Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2014. Learning Layouts for Single-PageGraphic Designs. IEEE Transactions on Visualization and Computer Graphics, Vol. 20, 8 (2014), 1200--1213. https://doi.org/10.1109/TVCG.2014.48

Digital Library

[26]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. dtextquotesingle Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024--8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

Digital Library

[27]

Akshay Gadi Patil, Omri Ben-Eliezer, Or Perel, and Hadar Averbuch-Elor. 2020. READ: Recursive Autoencoders for Document Layout Generation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2020), 2316--2325.

[28]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021. High-Resolution Image Synthesis with Latent Diffusion Models. arxiv: 2112.10752 [cs.CV]

[29]

Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. 2018. Adversarially learned one-class classifier for novelty detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3379--3388.

[30]

Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020).

[31]

Jun Wei, Shuhui Wang, and Qingming Huang. 2020. F3Net: fusion, feedback and focus for salient object detection. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 12321--12328.

[32]

Kota Yamaguchi. 2021. CanvasVAE: Learning to Generate Vector Graphic Documents. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 5461--5469. https://doi.org/10.1109/ICCV48922.2021.00543

[33]

Xuyong Yang, Tao Mei, Ying-Qing Xu, Yong Rui, and Shipeng Li. 2016. Automatic Generation of Visual-Textual Presentation Layout. ACM Trans. Multimedia Comput. Commun. Appl., Vol. 12, 2, Article 33 (feb 2016), 22 pages. https://doi.org/10.1145/2818709

Digital Library

[34]

Peiying Zhang, Chenhui Li, and Changbo Wang. 2020. Smarttext: Learning To Generate Harmonious Textual Layout Over Natural Image. In 2020 IEEE International Conference on Multimedia and Expo (ICME). 1--6. https://doi.org/10.1109/ICME46284.2020.9102780

[35]

Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson W. H. Lau. 2019. Content-Aware Generative Modeling of Graphic Design Layouts. ACM Trans. Graph., Vol. 38, 4, Article 133 (jul 2019), 15 pages. https://doi.org/10.1145/3306346.3322971

Digital Library

[36]

Min Zhou, Chenchen Xu, Ye Ma, Tiezheng Ge, Yuning Jiang, and Weiwei Xu. 2022. Composition-aware Graphic Layout GAN for Visual-Textual Presentation Designs. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, Lud De Raedt (Ed.). International Joint Conferences on Artificial Intelligence Organization, 4995--5001. https://doi.org/10.24963/ijcai.2022/692 AI and Arts.

[37]

Yibo Zhou. 2022. Rethinking reconstruction autoencoder-based out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7379--7387.

Cited By

Li YChen JBai YCheng JLei JSerra ESpezzano F(2024)Design Element Aware Poster Layout GenerationProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679557(1296-1305)Online publication date: 21-Oct-2024
https://doi.org/10.1145/3627673.3679557
Yang YChen CLi T(2024)Automated construction site layout design system for prefabricated buildings using transformer based conditional GANAdvanced Engineering Informatics10.1016/j.aei.2024.10288562:PDOnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.aei.2024.102885

Index Terms

Two-stage Content-Aware Layout Generation for Poster Designs
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
2. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods

Recommendations

Content-aware generative modeling of graphic design layouts

Layout is fundamental to graphic designs. For visual attractiveness and efficient communication of messages and ideas, graphic design layouts often have great variation, driven by the contents to be presented. In this paper, we study the problem of ...
PosterLlama: Bridging Design Ability of Language Model to Content-Aware Layout Generation
Computer Vision – ECCV 2024
Abstract
Visual layout plays a critical role in graphic design fields such as advertising, posters, and web UI design. The recent trend toward content-aware layout generation through generative models has shown promise, yet it often overlooks the semantic ...
GTLayout: Learning General Trees for Structured Grid Layout Generation
Computational Visual Media
Abstract
Structured grid layouts are preferable in many scenarios of 2D visual content creation since their structures facilitate further layout editing. Multiple geometry-based methods can effectively create structured grid layouts but require user-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
349
Total Downloads

Downloads (Last 12 months)152
Downloads (Last 6 weeks)3

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li YChen JBai YCheng JLei JSerra ESpezzano F(2024)Design Element Aware Poster Layout GenerationProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679557(1296-1305)Online publication date: 21-Oct-2024
https://doi.org/10.1145/3627673.3679557
Yang YChen CLi T(2024)Automated construction site layout design system for prefabricated buildings using transformer based conditional GANAdvanced Engineering Informatics10.1016/j.aei.2024.10288562:PDOnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.aei.2024.102885

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten