Abstract
In graphic design, layout is a result of the interaction between the design elements in the foreground and background images. However, prevalent research focuses on enhancing the quality of layout generation algorithms, overlooking the interaction and controllability that are essential for designers when applying these methods in real-world situations. This paper proposes a user-centered layout design system, Iris, which provides designers with an interactive environment to expedite the workflow, and this environment encompasses the features of user-constraint specification, layout generation, custom editing, and final rendering. To satisfy the multiple constraints specified by designers, we introduce a novel generation model, multi-constraint LayoutVQ-VAE, for advancing layout generation under intra- and inter-domain constraints. Qualitative and quantitative experiments on our proposed model indicate that it outperforms or is comparable to prevalent state-of-the-art models in multiple aspects. User studies on Iris further demonstrate that the system significantly enhances design efficiency while achieving human-like layout designs.
摘要
在平面设计中,布局是前景设计元素和背景图像相互作用的结果。然而,现有的研究主要集中在提高布局生成算法性能上,忽略设计师在现实世界中应用这些方法时所必需的交互性和可控性。本文提出一个以用户为中心的布局设计系统Iris,它为设计师提供了一个交互式的环境加快工作流程。该环境支持用户约束输入、布局生成、自定义编辑和布局渲染。为满足设计师指定的多种约束,引入一种新的生成模型——多约束 LayoutVQ-VAE,以推进在域内和域间多种条件约束下的布局生成。对所提模型进行定性和定量实验。实验结果表明,该模型在多个方面的表现优于目前最先进的模型或可与之相媲美。对Iris系统的用户研究进一步表明,该系统在显著提高设计效率的同时,也实现了接近人类设计师的布局设计。
Similar content being viewed by others
Data availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
Arroyo DM, Postels J, Tombari F, 2021. Variational Transformer networks for layout generation. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.13637–13647. https://doi.org/10.1109/CVPR46437.2021.01343
Ba JL, Kiros JR, Hinton GE, 2016. Layer normalization. https://arxiv.org/abs/1607.06450
Bangor A, Kortum P, Miller J, 2009. Determining what individual SUS scores mean: adding an adjective rating scale. J Usabil Stud, 4(3):114–123.
Cao YN, Ma Y, Zhou M, et al., 2022. Geometry aligned variational Transformer for image-conditioned layout generation. Proc 30th ACM Int Conf on Multimedia, p.1561–1571. https://doi.org/10.1145/3503161.3548332
Dayama NR, Todi K, Saarelainen T, et al., 2020. GRIDS: interactive layout design with integer programming. Proc CHI Conf on Human Factors in Computing Systems, p.1–13. https://doi.org/10.1145/3313831.3376553
Deka B, Huang ZF, Franzen C, et al., 2017. Rico: a mobile App dataset for building data-driven design applications. Proc 30th Annual ACM Symp on User Interface Software and Technology, p.845–854. https://doi.org/10.1145/3126594.3126651
Devlin J, Chang MW, Lee K, et al., 2019. BERT: pre-training of deep bidirectional Transformers for language understanding. Proc Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p.4171–4186.
Dosovitskiy A, Beyer L, Kolesnikov A, et al., 2021. An image is worth 16×16 words: Transformers for image recognition at scale. Proc 9th Int Conf on Learning Representations.
Guo SN, Jin ZC, Sun FL, et al., 2021. Vinci: an intelligent graphic design system for generating advertising posters. Proc CHI Conf on Human Factors in Computing Systems, Article 577. https://doi.org/10.1145/3411764.3445117
Gupta K, Lazarow J, Achille A, et al., 2021. LayoutTransformer: layout generation and completion with self-attention. Proc IEEE/CVF Int Conf on Computer Vision, p.984–994. https://doi.org/10.1109/ICCV48922.2021.00104
Hart SG, Staveland LE, 1988. Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv Psychol, 52:139–183. https://doi.org/10.1016/S0166-4115(08)62386-9
He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.770–778. https://doi.org/10.1109/CVPR.2016.90
Heusel M, Ramsauer H, Unterthiner T, et al., 2017. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Proc 30th Int Conf on Neural Information Processing Systems, p.6626–6637.
Hsu H, He XT, Peng YX, et al., 2023. PosterLayout: a new benchmark and approach for content-aware visual-textual presentation layout. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.6018–6026. https://doi.org/10.1109/CVPR52729.2023.00583
Hui MD, Zhang ZZ, Zhang XY, et al., 2023. Unifying layout generation with a decoupled diffusion model. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.1942–1951. https://doi.org/10.1109/CVPR52729.2023.00193
Inoue N, Kikuchi K, Simo-Serra E, et al., 2023. LayoutDM: discrete diffusion model for controllable layout generation. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.10167–10176. https://doi.org/10.1109/CVPR52729.2023.00980
Jacobs C, Li W, Schrier E, et al., 2003. Adaptive grid-based document layout. ACM Trans Graph, 22(3):838–847. https://doi.org/10.1145/882262.882353
Jiang ZY, Sun SZ, Zhu JH, et al., 2022. Coarse-to-fine generative modeling for graphic layouts. Proc 36th AAAI Conf on Artificial Intelligence, p.1096–1103. https://doi.org/10.1609/aaai.v36i1.19994
Jiang ZY, Guo JQ, Sun SZ, et al., 2023. LayoutFormer++: conditional graphic layout generation via constraint serialization and decoding space restriction. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.18403–18412. https://doi.org/10.1109/CVPR52729.2023.01765
Jing QZ, Zhou TT, Tsang Y, et al., 2023. Layout generation for various scenarios in mobile shopping applications. Proc CHI Conf on Human Factors in Computing Systems, Article 130. https://doi.org/10.1145/3544548.3581446
Kaiser L, Bengio S, Roy A, et al., 2018. Fast decoding in sequence models using discrete latent variables. Proc 35th Int Conf on Machine Learning, p.2395–2404.
Kikuchi K, Simo-Serra E, Otani M, et al., 2021. Constrained graphic layout generation via latent optimization. Proc 29th ACM Int Conf on Multimedia, p.88–96. https://doi.org/10.1145/3474085.3475497
Kong X, Jiang L, Chang HW, et al., 2022. BLT: bidirectional layout transformer for controllable layout generation. Proc 17th European Conf on Computer Vision, p.474–490. https://doi.org/10.1007/978-3-031-19790-1_29
Li JN, Yang JM, Hertzmann A, et al., 2019. LayoutGAN: generating graphic layouts with wireframe discriminators. Proc 7th Int Conf on Learning Representations.
Li JN, Yang JM, Zhang JM, et al., 2021. Attribute-conditioned layout GAN for automatic graphic design. IEEE Trans Vis Comput Graph, 27(10):4039–4048. https://doi.org/10.1109/TVCG.2020.2999335
Lin TY, Dollár P, Girshick R, et al., 2017. Feature pyramid networks for object detection. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2117–2125. https://doi.org/10.1109/CVPR.2017.106
O’Donovan P, Agarwala A, Hertzmann A, 2014. Learning layouts for single-page graphic designs. IEEE Trans Vis Comput Graph, 20(8):1200–1213. https://doi.org/10.1109/TVCG.2014.48
Paszke A, Gross S, Massa F, et al., 2019. PyTorch: an imperative style, high-performance deep learning library. Proc 32nd Int Conf on Neural Information Processing Systems, p.8024–8035.
Schrier E, Dontcheva M, Jacobs C, et al., 2008. Adaptive layout for dynamically aggregated documents. Proc 13th Int Conf on Intelligent User Interfaces, p.99–108. https://doi.org/10.1145/1378773.1378787
van den Oord A, Vinyals O, Kavukcuoglu K, 2017. Neural discrete representation learning. Proc 30th Int Conf on Neural Information Processing Systems, p.6306–6315.
Vaswani A, Shazeer N, Parmar N, et al., 2017. Attention is all you need. Proc 30th Int Conf on Neural Information Processing Systems, p.5998–6008.
Xu CC, Zhou M, Ge TZ, et al., 2023. Unsupervised domain adaption with pixel-level discriminator for image-aware layout generation. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.10114–10123. https://doi.org/10.1109/CVPR52729.2023.00975
You WT, Jiang H, Yang ZY, et al., 2020. Automatic synthesis of advertising images according to a specified style. Front Inform Technol Electron Eng, 21(10):1455–1466. https://doi.org/10.1631/FITEE.1900367
Zheng XR, Qiao XT, Cao Y, et al., 2019. Content-aware generative modeling of graphic design layouts. ACM Trans Graph, 38(4):133. https://doi.org/10.1145/3306346.3322971
Zhong X, Tang JB, Yepes AJ, 2019. PubLayNet: largest dataset ever for document layout analysis. Proc Int Conf on Document Analysis and Recognition, p.1015–1022. https://doi.org/10.1109/ICDAR.2019.00166
Zhou M, Xu CC, Ma Y, et al., 2022. Composition-aware graphic layout GAN for visual-textual presentation designs. Proc 31st Int Joint Conf on Artificial Intelligence, p.4995–5001.
Author information
Authors and Affiliations
Contributions
Liuqing CHEN, Qianzhi JING, and Yixin TSANG designed the research. Qianzhi JING and Yixin TSANG processed the data. Liuqing CHEN, Qianzhi JING, and Yixin TSANG drafted the paper. Liuqing CHEN and Tingting ZHOU revised and finalized the paper.
Corresponding author
Ethics declarations
All the authors declare that they have no conflict of interest.
Additional information
Project supported by the Alibaba–Zhejiang University Joint Research Institute of Frontier Technologies, China and the Zhejiang–Singapore Innovation and AI Joint Research Lab, China
List of supplementary materials
1 Examples for poster and magazine layout generation
2 Case study with Iris and Midjourney
3 Leveraging Midjourney for poster layout generation
Fig. S1 Examples for PDCard and magazine graphic layout design by Iris
Fig. S2 Multi-constraint poster layout generation comparison
Table S1 Prompts used and the corresponding generation results during the six iterations of layout generation using Midjourney
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Chen, L., Jing, Q., Tsang, Y. et al. Iris: a multi-constraint graphic layout generation system. Front Inform Technol Electron Eng 25, 968–987 (2024). https://doi.org/10.1631/FITEE.2300312
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2300312