Abstract
Synthesizing realistic 3D indoor scenes is a challenging task that traditionally relies on manual arrangement and annotation by expert designers. Recent advances in autoregressive models have automated this process, but they often lack semantic understanding of the relationships and hierarchies present in real-world scenes, yielding limited performance. In this paper, we propose Forest2Seq, a framework that formulates indoor scene synthesis as an order-aware sequential learning problem. Forest2Seq organizes the inherently unordered collection of scene objects into structured, ordered hierarchical scene trees and forests. By employing a clustering-based algorithm and a breadth-first traversal, Forest2Seq derives meaningful orderings and utilizes a transformer to generate realistic 3D scenes autoregressively. Experimental results on standard benchmarks demonstrate Forest2Seq’s superiority in synthesizing more realistic scenes compared to top-performing baselines, with significant improvements in FID and KL scores. Our additional experiments for downstream tasks and ablation studies also confirm the importance of incorporating order as a prior in 3D scene generation.
Q. Sun and H. Zhou—Equal contributions; Work carried out at SFU by Hang.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
https://www.ikea.com/pt/en/customer-service/services/planning-consultation
Bengio, Y., Ducharme, R., Vincent, P.: A neural probabilistic language model. In: NeurIPS (2000)
Bundy, A., Wallen, L.: Breadth-first search. In: Catalogue of Artificial Intelligence Tools (1984)
Chang, A., et al.: Matterport3D: learning from RGB-D data in indoor environments. In: 3DV (2017)
Chang, H., et al.: Muse: text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 (2023)
Chang, H., Zhang, H., Jiang, L., Liu, C., Freeman, W.T.: MaskGiT: masked generative image transformer. In: CVPR (2022)
Chaudhuri, S., Kalogerakis, E., Giguere, S., Funkhouser, T.: Attribit: content creation with semantic attributes. In: UIST (2013)
Currey, A., Heafield, K.: Unsupervised source hierarchies for low-resource neural machine translation. In: ACL (2018)
Deitke, M., et al.: ProcTHOR: large-scale embodied AI using procedural generation. In: NeurIPS (2022)
Devaranjan, J., Kar, A., Fidler, S.: Meta-sim2: unsupervised learning of scene structure for synthetic data generation. In: ECCV (2020)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019)
Dosovitskiy, Aet al.: An image is worth 16\(\times \)16 words: transformers for image recognition at scale. In: ICLR (2021)
Eriguchi, A., Hashimoto, K., Tsuruoka, Y.: Tree-to-sequence attentional neural machine translation. In: ACL (2016)
Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: CVPR (2021)
Ester, M., Kriegel, H.P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: KDD (1996)
Feng, W., et al.: LayoutGPT: compositional visual planning and generation with large language models. arXiv preprint arXiv:2305.15393 (2023)
Fisher, M., Ritchie, D., Savva, M., Funkhouser, T., Hanrahan, P.: Example-based synthesis of 3D object arrangements. ACM TOG 31, 1–11 (2012)
Fisher, M., Savva, M., Li, Y., Hanrahan, P., Nießner, M.: Activity-centric scene synthesis for functional 3D scene modeling. ACM TOG 34, 1–13 (2015)
Fu, H., et al.: 3D-FRONT: 3D furnished rooms with layouts and semantics. In: CVPR (2021)
Fu, H., et al.: 3D-FUTURE: 3D furniture shape with texture. IJCV 129, 3313–3337 (2021)
Fu, Z., et al.: Decoder-only or encoder-decoder? interpreting language model as a regularized encoder-decoder. arXiv preprint arXiv:2304.04052 (2023)
Gani, H., Naseer, M., Yaqub, M.: How to train vision transformer on small-scale datasets? In: BMVC (2022)
Gao, L., Sun, J.M., Mo, K., Lai, Y.K., Guibas, L.J., Yang, J.: SceneHGN: hierarchical graph networks for 3d indoor scene generation with fine-grained geometry. IEEE TPAMI 45, 8902–8919 (2023)
Gu, S., et al.: Vector quantized diffusion model for text-to-image synthesis. In: CVPR (2022)
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: CVPR (2022)
Henderson, P., Subr, K., Ferrari, V.: Automatic generation of constrained furniture layouts. arXiv preprint arXiv:1711.10939 (2017)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS (2017)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. (1997)
Iyer, S., Konstas, I., Cheung, A., Zettlemoyer, L.: Summarizing source code using a neural attention model. In: ACL (2016)
Jiang, Y., Lim, M., Saxena, A.: Learning object arrangements in 3D scenes using human context. arXiv preprint arXiv:1206.6462 (2012)
Kar, A., et al.: Meta-sim: learning to generate synthetic datasets. In: ICCV (2019)
Ke, G., He, D., Liu, T.Y.: Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006.15595 (2020)
Koh, J.Y., et al.: Simple and effective synthesis of indoor 3d scenes. In: AAAI (2023)
Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., Teh, Y.W.: Set transformer: a framework for attention-based permutation-invariant neural networks. In: ICML (2019)
Leimer, K., Guerrero, P., Weiss, T., Musialski, P.: Layoutenhancer: generating good indoor layouts from imperfect data. In: SIGGRAPH Asia (2022)
Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: ACL (2020)
Li, M., et al.: GRAINS: generative recursive autoencoders for indoor scenes. ACM TOG 38, 1–16 (2019)
Li, R., et al.: StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161 (2023)
Liu, J., et al.: CLIP-Layout: Style-consistent indoor scene synthesis with semantic furniture embedding. arXiv preprint arXiv:2303.03565 (2023)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019)
Ma, R., Li, H., Zou, C., Liao, Z., Tong, X., Zhang, H.: Action-driven 3D indoor scene evolution. ACM TOG 35, 173-1 (2016)
Ma, R., et al.: Language-driven synthesis of 3D scenes from scene databases. ACM TOG 37, 1–16 (2018)
Merrell, P., Schkufza, E., Li, Z., Agrawala, M., Koltun, V.: Interactive furniture layout using interior design guidelines. ACM TOG 30, 1–10 (2011)
OpenAI: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
Ouyang, L., et al.: Training language models to follow instructions with human feedback. In: NeurIPS (2022)
Para, W.R., Guerrero, P., Mitra, N., Wonka, P.: COFS: controllable furniture layout synthesis. In: SIGGRAPH (2023)
Paschalidou, D., Kar, A., Shugrina, M., Kreis, K., Geiger, A., Fidler, S.: ATISS: autoregressive transformers for indoor scene synthesis. In: NeurIPS (2021)
Patil, A.G., Patil, S.G., Li, M., Fisher, M., Savva, M., Zhang, H.: Advances in data-driven analysis and synthesis of 3D indoor scenes. Comput. Graph Forum (2023)
Qi, S., Zhu, Y., Huang, S., Jiang, C., Zhu, S.C.: Human-centric indoor scene synthesis using stochastic grammar. In: CVPR (2018)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. In: ICML (2019)
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union: a metric and a loss for bounding box regression. In: CVPR (2019)
Ritchie, D., Wang, K., Lin, Y.a.: Fast and flexible indoor scene synthesis via deep convolutional generative models. In: CVPR (2019)
Rubenstein, P.K., et al.: AudioPaLM: a large language model that can speak and listen. arXiv preprint arXiv:2306.12925 (2023)
Savva, M., Chang, A.X., Hanrahan, P., Fisher, M., Nießner, M.: PiGraphs: learning interaction snapshots from observations. ACM TOG 30, 1–10 (2016)
Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 (2018)
Socher, R., Lin, C.C.Y., Ng, A.Y., Manning, C.D.: Parsing natural scenes and natural language with recursive neural networks. In: ICML (2011)
Straub, J., et al.: The replica dataset: a digital replica of indoor spaces. arXiv preprint arXiv:1906.05797 (2019)
Tang, J., Nie, Y., Markhasin, L., Dai, A., Thies, J., Nießner, M.: DiffuScene: scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. arXiv preprint arXiv:2303.14207 (2023)
Tang, K., Zhang, H., Wu, B., Luo, W., Liu, W.: Learning to compose dynamic tree structures for visual contexts. In: CVPR (2019)
Vaswani, A., et al.: Attention is all you need. In: NeurIPS (2017)
Vinyals, O., Bengio, S., Kudlur, M.: Order matters: sequence to sequence for sets. In: ICLR (2016)
Wang, K., Lin, Y.A., Weissmann, B., Savva, M., Chang, A.X., Ritchie, D.: PlanIT: planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM TOG 38, 1–15 (2019)
Wang, X., Yeshwanth, C., Nießner, M.: SceneFormer: indoor scene generation with transformers. In: 3DV (2021)
Wei, Q.A., et al.: LEGO-Net: learning regular rearrangements of objects in rooms. In: CVPR (2023)
Weiss, T., et al.: Fast and scalable position-based layout synthesis. IEEE TVCG 25, 3231–3243 (2018)
Xu, K., Wu, L., Wang, Z., Feng, Y., Witbrock, M., Sheinin, V.: Graph2seq: graph to sequence learning with attention-based neural networks. arXiv preprint arXiv:1804.00823 (2018)
Yu, L.F., Yeung, S.K., Tang, C.K., Terzopoulos, D., Chan, T.F., Osher, S.J.: Make It Home: automatic optimization of furniture arrangement. ACM TOG 30 (2011)
Yu, L.F., Yeung, S.K., Terzopoulos, D.: The ClutterPalette: an interactive tool for detailing indoor scenes. IEEE TVCG 22, 1138–1148 (2015)
Zaheer, M., Kottur, S., Ravanbhakhsh, S., Póczos, B., Salakhutdinov, R., Smola, A.J.: Deep sets. In: NeurIPS (2017)
Zellers, R., Yatskar, M., Thomson, S., Choi, Y.: Neural motifs: scene graph parsing with global context. In: CVPR (2018)
Zhai, G., Örnek, E.P., Wu, S.C., Di, Y., Tombari, F., Navab, N., Busam, B.: CommonScenes: generating commonsense 3D indoor scenes with scene graphs. arXiv preprint arXiv:2305.16283 (2023)
Zhao, X., Hu, R., Guerrero, P., Mitra, N., Komura, T.: Relationship templates for creating scene variations. ACM TOG 35, 1–13 (2016)
Zhao, Y., et al.: Luminous: indoor scene generation for embodied ai challenges. In: NeurIPSW (2021)
Zhou, Y., While, Z., Kalogerakis, E.: SceneGraphNet: neural message passing for 3D indoor scene augmentation. In: CVPR (2019)
Acknowledgements
The authors would like to thank Nathan Yan (Cornell) and Prof. Jing Liao (CityU) for useful comments and discussions. This work is supported by National Natural Science Foundation of China under Contract 62021001 and the Youth Innovation Promotion Association CAS. It was also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sun, Q., Zhou, H., Zhou, W., Li, L., Li, H. (2025). FOREST2SEQ: Revitalizing Order Prior for Sequential Indoor Scene Synthesis. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15083. Springer, Cham. https://doi.org/10.1007/978-3-031-72698-9_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-72698-9_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72697-2
Online ISBN: 978-3-031-72698-9
eBook Packages: Computer ScienceComputer Science (R0)