Abstract
We present a vision-based algorithm that uses spatio-temporal satellite imagery, pattern recognition, procedural modeling, and deep learning to perform tree localization in urban settings. Our method resolves two primary challenges. First, automated city-scale tree localization at high accuracy typically requires significant acquisition/user intervention. Second, vegetation-index segmentation methods from satellites require manual thresholding, which varies across geographic areas, and are not robust across cities with varying terrain, geometry, altitude, and canopy. In our work, we compensate for the lack of visual detail by using satellite snapshots across twelve months and segment cities into various vegetation clusters. Then, we use multiple GAN-based networks to plant trees by recognizing placement patterns inside segmented regions procedurally. We present comprehensive experiments over four cities (Chicago, Austin, Indianapolis, Lagos), achieving tree count accuracies of 87–97%. Finally, we show that the knowledge accumulated from each model (trained on a particular city) can be transferred to a different city.











Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., Kim, B.: Sanity checks for saliency maps. In: NeurIPS, NIPS’18, pp. 9525–9536 (2018)
Aliaga, D.G., Vanegas, C., Lei, M., Niyogi, D.: Visualization-based decision tool for urban meteorological modeling. Environ. Plann. B. Plann. Des. 40(2), 271–288 (2013)
Arief, H.A., Strand, G.H., Tveite, H., Indahl, U.G.: Land cover segmentation of airborne lidar data using stochastic atrous network. Remote Sens. 10(6), (2018)
Benes, B., Massih, M.A., Jarvis, P., Aliaga, D.G., Vanegas, C.A.: Urban ecosystem design. In: Symposium on Interactive 3D Graphics and Games, I3D ’11, pp. 167–174. Association for Computing Machinery (2011)
Bovik, A.: A visual information fidelity approach to video quality assesment (2005)
Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV (2018)
Chicago Department of Transportation: Street and site plan design standards (2007). https://www.chicago.gov/dam/city/depts/cdot/StreetandSitePlanDesignStandards407.pdf
City of Austin Texas: Downtown Tree Inventory (2013). https://data.austintexas.gov/Locations-and-Maps/Downtown-Tree-Inventory-2013
City of Indianapolis/Marion, Indiana: Code of ordinances—chapter 701—trees and flora (2021). https://library.municode.com/in/indianapolis-marion_county/codes/code_of_ordinances
City of New York: NYC open data map tiles (2021). https://maps.nyc.gov/tiles
Earth Observing System (EOS), NASA: The Ice, Cloud, and Land Elevation Satellite-2 (2021). https://icesat-2.gsfc.nasa.gov/icesat-2-data
Food and Agriculture Organization of the United Nations: Global forest resources assessment country report series—Nigeria. Tech. rep, United Nations (2010)
Guo, J., Jiang, H., Benes, B., Deussen, O., Zhang, X., Lischinski, D., Huang, H.: Inverse procedural modeling of branching structures by inferring l-systems. ACM Trans. Graph. 39(5) (2020)
Hahsler, M., Piekenbrock, M., Doran, D.: dbscan: Fast density-based clustering with R. J. Stat. Softw. 91(1), 1–30 (2019)
Hojas-Gascon, L., Eva, H.: Field guide for forest mapping with high resolution satellite data. monitoring deforestation and forest degradation in the context of the un-redd programme. the tanzania redd + initiative (2014). https://doi.org/10.2788/657954
Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q.V., Adam, H.: Searching for mobilenetv3. In: ICCV (2019)
Hu, Q., Yang, B., Khalid, S., Xiao, W., Trigoni, N., Markham, A.: Towards semantic segmentation of urban-scale 3d point clouds: a dataset, benchmarks and challenges (2020). https://doi.org/10.48550/ARXIV.2009.03137
Huang, S., Tang, L., Hupy, J., Wang, Y., Shao, G.: A commentary review on the use of normalized difference vegetation index (ndvi) in the era of popular remote sensing. J. For. Res. 32 (2020)
Huang, Z., Arian, A., Yuan, Y., Chiu, Y.C.: Using conditional generative adversarial nets and heat maps with simulation-accelerated training to predict the spatiotemporal impacts of highway incidents. Transp. Res. Rec. 2674(8), 836–849 (2020)
Inc., G.: Google Earth (2021). http://earth.google.com
Japan Aerospace Exploration Agency—JAXA: JAXA (2021). https://global.jaxa.jp
Jiang, Z., Huete, A., Chen, J., Chen, Y., Li, J., Yan, G., Zou, Y.: Analysis of ndvi and scaled difference vegetation index retrievals of vegetation fraction. Remote Sens. Environ. 101, 366–378 (2006)
Keren, S.: Modeling tree species count data in the understory and canopy layer of two mixed old-growth forests in the dinaric region. Forests 11(5) (2020)
Kim, J.S., Kavak, H., Crooks, A.: Procedural city generation beyond game development. SIGSPATIAL Special 10(2), 34–41 (2018)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NeurIPS, vol. 25 (2012)
Lee, S., Park, S., Son, S., Han, J., Kim, S., Kim, J.: Land cover segmentation of aerial imagery using segnet. In: Earth Resources and Environmental Remote Sensing/GIS Applications X, vol. 11156, pp. 313–318. Intl. Society for Optics and Photonics, SPIE (2019)
Li, B., Kałużny, J., Klein, J., Michels, D.L., Pałubicki, W., Benes, B., Pirk, S.: Learning to reconstruct botanical trees from single images. ACM Trans. Graph. 40(6) (2021)
Lian, D., Li, J., Zheng, J., Luo, W., Gao, S.: Density map regression guided detection network for rgb-d crowd counting and localization. In: CVPR, pp. 1821–1830 (2019). https://doi.org/10.1109/CVPR.2019.00192
Liu, L., Chen, J., Wu, H., Li, G., Li, C., Lin, L.: Cross-modal collaborative representation learning and a large-scale rgbt benchmark for crowd counting. In: CVPR, pp. 4823–4833 (2021)
Liu, Y., Guo, J., Benes, B., Deussen, O., Zhang, X., Huang, H.: Treepartnet: Neural decomposition of point clouds for 3d tree reconstruction. ACM Trans. Graph. 40(6) (2021)
Lu, D., Weng, Q.: A survey of image classification methods and techniques for improving classification performance. Int. J. Remote Sens. 28(5), 823–70 (2007)
Maggiori, E., Tarabalka, Y., Charpiat, G., Alliez, P.: The inria aerial image labeling benchmark. In: 2017 IEEE Intl. Geoscience and Remote Sensing Symposium (IGARSS) (2017)
Martinovic, A., Van Gool, L.: Bayesian grammar learning for inverse procedural modeling. In: CVPR, pp. 201–208 (2013)
McBride, J.: Mapping chicago area urban tree canopy using color infrared imagery. Ph.D. thesis, Lund University, Lund, Sweden (2011)
Musialski, P., Wonka, P., Aliaga, D.G., Wimmer, M., Van Gool, L., Purgathofer, W.: A survey of urban reconstruction. In: Computer Graphics Forum, vol. 32, pp. 146–177. Wiley Online Library (2013)
Niese, T., Pirk, S., Albrecht, M., Benes, B., Deussen, O.: Procedural urban forestry. ACM Trans. Graph. 41(2) (2022)
Nowak, D., Bodine, A.R., Hoehn, R.E., Edgar, C., Hartel, D., Lister, T., Brandeis, T.: Austin’s urban forest (2016)
Nowak, D., Heisler, N.: Air quality effects of urban trees and parks. Research Series Monograph. National Recreation and Parks Association Research Series Monograph, Ashburn, VA (2010). https://www.fs.usda.gov/treesearch/pubs/52881
Open Street Map Contributors: Planet Dump (2021). https://planet.osm.org. https://www.openstreetmap.org
Paula, P., Gregory, M., James, S., Kelaine, V., Qingfu, X.: City of Indianapolis, Indiana Municipal Forest Resource Analysis. Tech. rep. Center for Urban Forest Research USDA Forest Service (2008)
Planet: Planet Explorer (2021). http://planet.com/explorer
Potapov, P., Li, X., Hernandez-Serna, A., Tyukavina, A., Hansen, M., Kommareddy, A., Pickens, A., Turubanova, S., Tang, H., Silva, C.E., Armston, J., Dubayah, R., Blair, J.B., Hofton, M.: Mapping global forest canopy height through integration of gedi and landsat data. Remote Sens. Environ. 253, 112–165 (2021)
Pretzsch, H., Biber, P., Uhl, E., Dahlhausen, J., Rötzer, T., Caldentey, J., Koike, T., van Con, T., Chavanne, A., Seifert, T., du Toit, B., Farnden, C., Pauleit, S.: Crown size and growing space requirement of common tree species in urban centres, parks, and forests. Urban For. Urban Greening 14(3), 466–479 (2015)
Prusinkiewicz, P., Lindenmayer, A.: The Algorithmic Beauty of Plants. Springer-Verlag, New York (1990)
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI 2015, pp. 234–241. Cham (2015)
Rußwurm, M., Körner, M.: Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images. In: CVPR, pp. 1496–1504 (2017)
Shen, B., Fang, S., Li, G.: Vegetation coverage changes and their response to meteorological variables from 2000 to 2009 in Naqu, Tibet, China. Can. J. Remote Sens. 40, 67–74 (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)
Tinchev, G., Nobili, S., Fallon, M.: Seeing the Wood for the Trees: Reliable Localization in Urban and Natural Environments (2018). https://doi.org/10.48550/ARXIV.1809.02846
Tree Care Industry Association (TCIA): ANSI A300 Standards: Tree, Shrub and Other Woody Plant Management—Standard Practices. American National Standards Institute (2017). https://www.tcia.org/TCIA/Build_Your_Business/A300_Standards/A300_Standards.aspx
Tree counting with high spatial-resolution satellite imagery based on deep neural networks. Ecol. Ind. 125, 107591 (2021)
Urban Design Division/Planning & Development Review Department: City of austin great streets (2012)
USDA Forest Service: iTree (2020). https://www.itreetools.org
Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Img. Proc. 13(4), 600–612 (2004)
Wang, Z., Simoncelli, E., Bovik, A.: Multiscale structural similarity for image quality assessment. Conf. Record Asilomar Conf. Signals Syst. Comp. 2 (2003)
Weier, J., Herring, D.: Measuring vegetation (ndvi and evi (2000). https://earthobservatory.nasa.gov/features/MeasuringVegetation
Yang, S.D., Su, H.T., Hsu, W.H., Chen, W.C.: Class-agnostic few-shot object counting. In: IEEE/CVF WACV, pp. 870–878 (2021)
Zabelskyte, G., Kabisch, N., Stasiskiene, Z.: Patterns of urban green space use applying social media data: a systematic literature review. Land 11(2) (2022)
Zhang, L., Zhang, L., Mou, X., Zhang, D.: Fsim: A feature similarity index for image quality assessment. IEEE Trans. Img. Proc. 20(8) (2011)
Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR, pp. 6230–6239 (2017)
Acknowledgements
This research was funded in part by National Science Foundation grant #10001387, Functional Proceduralization of 3D Geometric Models and by the Foundation for Food and Agriculture Research Grant ID: 602757. We thank the Integrated Digital Forestry Initiative (iDIF) at Purdue University for their partial support. We also thank NSF grant #1835739 “U-Cube: A Cyberinfrastructure for Unified and Ubiquitous Urban Canopy Parameterization” and NSF grant #2106717 “Deep Generative Modeling for Urban and Archaeological Recovery.”
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors claim and announce no conflict of interest with any entity or party in relation to this work.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Firoze, A., Benes, B. & Aliaga, D. Urban tree generator: spatio-temporal and generative deep learning for urban tree localization and modeling. Vis Comput 38, 3327–3339 (2022). https://doi.org/10.1007/s00371-022-02526-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-022-02526-x