Abstract
We show how a Transformer can encode hierarchical tree-like string structures by introducing a new deep learning-based framework for generating 3D biological tree models represented as Lindenmayer system (L-system) strings. L-systems are string-rewriting procedural systems that encode tree topology and geometry. L-systems are efficient, but creating the production rules is one of the most critical problems precluding their usage in practice. We substitute the procedural rules creation with a deep neural model. Instead of writing the rules, we train a deep neural model that produces the output strings. We train our model on 155k tree geometries that are encoded as L-strings, de-parameterized, and converted to a hierarchy of linear sequences corresponding to branches. An end-to-end deep learning model with an attention mechanism then learns the distributions of geometric operations and branches from the input, effectively replacing the L-system rewriting rule generation. The trained deep model generates new L-strings representing 3D tree models in the same way L-systems do by providing the starting string. Our model allows for the generation of a wide variety of new trees, and the deep model agrees with the input by 93.7% in branching angles, 97.2% in branch lengths, and 92.3% in an extracted list of geometric features. We also validate the generated trees using perceptual metrics showing 97% agreement with input geometric models.
Supplemental Material
Available for Download
Supplementary material
- 1984. Botanical tree image generation. IEEE Computer Graphics and Applications 4, 5 (1984), 10–34.Google ScholarDigital Library .
- 1988. Modeling plants with environment-sensitive automata. In Proceedings of the Ausgraph. 27–33.Google Scholar .
- 2021. Techniques for inferring context-free lindenmayer systems with genetic algorithm. Swarm and Evolutionary Computation 64 (2021), 100893.Google ScholarCross Ref .
- 2006. Pattern Recognition and Machine Learning. Vol. 4. Springer.Google ScholarDigital Library .
- 2010. A connection between partial symmetry and inverse procedural modeling. In Proceedings of the ACM SIGGRAPH 2010 Papers. 1–10.Google ScholarDigital Library .
- 2022. ProteinBERT: A universal deep-learning model of protein sequence and function. Bioinformatics 38, 8
(2022 ), 2102–2110.DOI: arXiv:https://academic.oup.com/bioinformatics/article-pdf/38/8/2102/4900 9610/btac020.pdf Google ScholarCross Ref . - 2019. Floor-sp: Inverse cad for floorplans by sequential room-wise shortest path. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2661–2670.Google ScholarCross Ref .
- 1988. Plant models faithful to botanical structure and development. ACM Siggraph Computer Graphics 22, 4 (1988), 151–158.Google ScholarDigital Library .
- 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv: 1810.04805. Retrieved from https://arxiv.org/abs/1810.04805Google Scholar .
- 2018. Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Proceedings of the AAAI Conference on Artificial Intelligence.Google ScholarCross Ref .
- 2019. AdTree: Accurate, detailed, and automatic modelling of laser-scanned trees. Remote Sensing 11, 18 (2019), 2074.Google ScholarCross Ref .
- 2018. InverseCSG: Automatic conversion of 3D models to CSG trees. ACM Transactions on Graphics 37, 6 (2018), 16 pages.
DOI: Google ScholarDigital Library . - 2018. Maskgan: Better text generation via filling in the_. arXiv:1801.07736. Retrieved from https://arxiv.org/abs/1801.07736Google Scholar .
- 2018. Evolving complete L-systems: Using genetic algorithms for the generation of realistic plants. In Proceedings of the Artificial Life and Intelligent Agents Symposium. Springer, 16–23.Google ScholarCross Ref .
- 2019. SDM-NET: Deep generative network for structured deformable mesh. ACM Transactions on Graphics 38, 6 (2019), 1–15.Google ScholarDigital Library .
- 2014. Generative adversarial nets. Advances in Neural Information Processing Systems 27 (2014), 2672–2680.Google Scholar .
- 1914. Grundrib der gasamten Gewässerkunde,band I: Flubkunde. kompendium of Hydrology 1 (1914), 265–278.Google Scholar .
- 1989. Voxel space automata: Modeling with stochastic growth processes in voxel space. Proceedings of the 16th annual Csonference on Computer Graphics and Interactive Techniques 23, 3 (1989), 175–184. Google ScholarDigital Library .
- 2020. Inverse procedural modeling of branching structures by inferring l-systems. ACM Transactions on Graphics 39, 5 (2020), 13 pages. Google ScholarDigital Library .
- 2018. Long text generation via adversarial training with leaked information. In Proceedings of the AAAI Conference on Artificial Intelligence.Google ScholarCross Ref .
- 2009. Physically guided animation of trees. In Proceedings of the Computer Graphics Forum. Wiley Online Library, 523–532.Google ScholarCross Ref .
- 2021. Fire in paradise: Mesoscale simulation of wildfires. ACM Transactions on Graphics 40, 4 (2021), 15 pages.
DOI: Google ScholarDigital Library . - 2017. Interactive modeling and authoring of climbing plants. 36, 2 (2017), 49–61.
DOI: arXiv:Google ScholarDigital Library . - 1997. Long short-term memory. Neural Computation 9, 8 (1997), 1735–1780.Google ScholarDigital Library .
- 1994. Strands, gravity and botanical tree imagery. In Proceedings of the Computer Graphics Forum. Wiley Online Library, 57–67.Google ScholarCross Ref .
- 2020. ShapeAssembly: Learning to generate programs for 3D shape structure synthesis. ACM Transactions on Graphics 39, 6 (2020), 20 pages.
DOI: Google ScholarDigital Library . - 2021. Highly accurate protein structure prediction with AlphaFold. Nature 596, 7873 (2021), 583–589.Google ScholarCross Ref .
- 2004. The l-system-based plant-modeling environment l-studio 4.0. In Proceedings of the 4th International Workshop on Functional-structural Plant Models. UMR AMAP Montpellier, France, 403–405.Google Scholar .
- 2014. Adam: A method for stochastic optimization. arXiv:1412.6980. Retrieved from https://arxiv.org/abs/1412.6980Google Scholar .
- Bosheng Li, Jacek Kałużny, Jonathan Klein, Dominik L. Michels, Wojtek Pałubicki, Bedrich Benes, and Sören Pirk. 2021. Learning to reconstruct botanical trees from single images. ACM Trans. Graph. 40, 6, Article 231 (December 2021), 15 pages. Google ScholarDigital Library
- Bosheng Li, Jonathan Klein, Dominik L. Michels, Bedrich Benes, Sören Pirk, and Wojtek Pałubicki. 2023. Rhizomorph: The coordinated function of shoots and roots. ACM Trans. Graph. 42, 4, Article 59 (August 2023), 16 pages. Google ScholarDigital Library
- 2017. Grass: Generative recursive autoencoders for shape structures. ACM Transactions on Graphics 36, 4 (2017), 1–14.Google ScholarDigital Library .
- 2018. A generative model for category text generation. Information Sciences 450 (2018), 301–315.Google ScholarDigital Library .
- 1968. Mathematical models for cellular interaction in development. Journal of Theoretical Biology Parts I and II, 18, 3 (1968), 280–315.Google ScholarCross Ref .
- 2012. TreeSketch: Interactive procedural modeling of trees on a tablet. In Proceedings of the SBIM Expressive. 107–120.Google Scholar .
- 2005. The FL-system: A functional l-system for procedural geometric modeling. The Visual Computer 21, 5 (2005), 329–339.Google ScholarDigital Library .
- 2014. Tree branching: Leonardo da vinci’s rule versus biomechanical models. PloS One 9, 4 (2014), e93535.Google ScholarCross Ref .
- Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy J. Mitra, and Leonidas J. Guibas. 2019. StructureNet: hierarchical graph networks for 3D shape generation. ACM Trans. Graph. 38, 6, Article 242 (December 2019), 19 pages. Google ScholarDigital Library
- 1996. Visual models of plants interacting with their environment. In SIGGRAPH ’96: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, NY, 397–410.
DOI: Google ScholarDigital Library . - 2020. Polygen: An autoregressive generative model of 3d meshes. In Proceedings of the International Conference on Machine Learning. PMLR, 7220–7229.Google Scholar .
- 1998. On genetic algorithms and lindenmayer systems. In Proceedings of the International Conference on Parallel Problem Solving from Nature. Springer, 335–344.Google ScholarCross Ref .
- 2007. Interactive design of botanical trees using freehand sketches and example-based editing. In Proceedings of the ACM SIGGRAPH 2007 Courses.Association for Computing Machinery, New York, NY, 26–es.
DOI: Google ScholarDigital Library . - 1986. Real time design and animation of fractal plants and trees. ACM SiGGRAPH Computer Graphics 20, 4 (1986), 55–64.
DOI: Google ScholarDigital Library . - 2009. Self-organizing tree models for image synthesis. ACM TOG 28, 3 (2009), 10 pages. Google ScholarDigital Library .
- 2016. Modeling plant life in computer graphics. In Proceedings of the ACM SIGGRAPH 2016 Courses. 180 pages.
DOI: Google ScholarDigital Library . - 2014. Windy trees: Computing stress response for developmental tree models. ACM Transactions on Graphics 33, 6 (2014), 11 pages.
DOI: Google ScholarDigital Library . - 2012. Plastic trees: Interactive self-adapting botanical tree models. ACM TOG 31, 4 (2012). Google ScholarDigital Library .
- 2021. ICTree: Automatic perceptual metrics for tree models. ACM Transaction on Graphics 40, 6 (2021), 15 pages.
DOI: Google ScholarDigital Library . - 1986. Graphical applications of l-systems. In Proceedings on Graphics Interface ’86/Vision Interface ’86.Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 247–253. Retrieved from http://dl.acm.org/citation.cfm?id=16564.16608Google ScholarDigital Library .
- 1993. Animation of plant development. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques. 351–360.Google ScholarDigital Library .
- 1990. Visualization of botanical structures and processes using parametric l-systems. In Proceedings of the Scientific Visualization and Graphics Simulation. 183–201.Google ScholarDigital Library .
- 1990. The Algorithmic Beauty of Plants. Springer–Verlag, New York.
With J. S. Hanan, F. D. Fracchia, D. R. Fowler, M. J. de Boer, and L. Mercer .Google ScholarDigital Library . - 2014. A survey on procedural modelling for virtual worlds. Computer Graphics Forum 33, 6 (2014), 31–50.
DOI: arXiv:Google ScholarDigital Library . - 1984. Plants, fractals, and formal languages. In Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, NY, 1–10.
DOI: Google ScholarDigital Library . - 2014. Inverse procedural modelling of trees. In Proceedings of the Computer Graphics Forum. Wiley Online Library, 118–131.Google ScholarDigital Library .
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (NIPS’14). MIT Press, Cambridge, MA, 3104–3112.Google Scholar
- 2012. Inverse design of urban procedural models. ACM Transactions on Graphics 31, 6 (2012), 1–11.Google ScholarDigital Library .
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Curran Associates Inc., Red Hook, NY, 6000–6010.Google Scholar
- 2018. Statistical modeling of the 3D geometry and topology of botanical trees. Computer Graphics Forum 37, 5 (2018), 185–198.Google ScholarCross Ref .
- 1995. Creation and rendering of realistic trees. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques.Association for Computing Machinery, New York, NY, 119–128.
DOI: Google ScholarDigital Library . - 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1, 2 (1989), 270–280.
DOI: Google ScholarDigital Library . - Fuzhang Wu, Dong-Ming Yan, Weiming Dong, Xiaopeng Zhang, and Peter Wonka. 2014. Inverse procedural modeling of facade layouts. ACM Trans. Graph. 33, 4, Article 121 (July 2014), 10 pages. Google ScholarDigital Library
- 2021. Do transformers really perform badly for graph representation?. In Proceedings of the Advances in Neural Information Processing Systems. , , , , and (Eds.), Vol. 34, Curran Associates, Inc., 28877–28888. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2021/file/f1c1592588411002af340cbaedd6fc33-Paper.pdfGoogle Scholar .
- Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J. Kim. 2019. Graph transformer networks. Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, Article 1073, 11983–11993.Google Scholar
- 2023. DeepTree: Modeling trees with situated latents. IEEE Transactions on Visualization and Computer Graphics01 (2023), 1–14. Google ScholarCross Ref .
Index Terms
- Latent L-systems: Transformer-based Tree Generator
Recommendations
Automatic Procedural Modeling of Tree Structures in Point Clouds Using Wavelets
3DV '13: Proceedings of the 2013 International Conference on 3D VisionWe present a method for discovering the structure of trees in 3D point clouds by linking wavelets with shape grammars. Given a range scan of a tree we find a grammar that can reproduce that tree, and others like it, with sub-voxel accuracy. The grammar ...
Generating vast varieties of realistic leaves with parametric 2Gmap L-systems
Creating realistic plants and trees require the ability to generate thousands of leaves with different shapes and textures for different given species. This paper presents an original method to generate large atlases of leaves with many details from a ...
L-system based interactive and lightweight web3D tree modeling
VRCAI '11: Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in IndustryVR applications on the Web demand lightweight virtual models and scenes. Among the existing tree modeling methods, the rule based modeling method, e.g. L-systems or automatons, should be the lightest. This paper presents an interactive approach for ...
Comments