Skip to main content

Advertisement

SpringerLink
Log in

CityCraft: 3D virtual city creation from a single image

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

This paper introduces a method to generate a three-dimensional (3D) virtual model of an imaginary city from a single street-view image to represent the appearance of the city in a given input photograph. The proposed approach differs from reconstruction approaches, which generate a city model by guessing the city name from the input photograph. In contrast, we use machine learning to identify where to generate the city, what to allocate in the city, and how to arrange the components. We employ generative adversarial networks and convolutional neural networks to create a terrain map and identify the components and styles that represent the virtual city appearance. We demonstrate that our system creates 3D virtual cities that are visually similar in terms of plausibility and naturalness to actual cities corresponding to input photographs from around the world. To the best of our knowledge, this is the first work to generate a city model including all general city components, including streets, buildings, and vegetation, to match the style of a single input image.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. https://terrain.party/.

  2. http://maps.stamen.com.

  3. https://www.esri.com/en-us/arcgis/products/esri-cityengine/overview.

References

  1. AlHalawani, S., Yang, Y.L., Wonka, P., Mitra, N.J.: What makes London work like London? Comput Graph Forum 33(5), 157–165. https://doi.org/10.1111/cgf.12441

  2. Aliaga, D.G., Demir, İ., Benes, B., Wand, M.: Inverse procedural modeling of 3D models for virtual worlds. In: ACM SIGGRAPH 2016 Courses, p. 16. ACM (2016)

  3. Aliaga, D.G., Vanegas, C.A., Beneš, B.: Interactive example-based urban layout synthesis.ACM Trans. Graph. 27(5), 160:1–160:10 (2008). https://doi.org/10.1145/1409060.1409113

  4. Argudo, O., Andujar, C., Chica, A., Guérin, E., Digne, J., Peytavie, A., Galin, E.: Coherent multi-layer landscape synthesis. Vis. Comput. 33(6–8), 1005–1015 (2017). https://doi.org/10.1007/s00371-017-1393-6

  5. Bao, F., Schwarz, M., Wonka, P.: Procedural facade variations from a single layout. ACM Trans. Graph. (TOG) 32(1), 8 (2013)

    Article  MATH  Google Scholar 

  6. Bauer, K.W.: City Planning for Civil Engineers, Environmental Engineers, and Surveyors. CRC Press, Boca Raton (2009)

    Book  Google Scholar 

  7. Bellotti, F., Berta, R., Cardona, R., De Gloria, A.: An architectural approach to efficient 3D urban modeling. Comput. Graph. 35(5), 1001–1012 (2011)

    Article  Google Scholar 

  8. Chen, G., Esch, G., Wonka, P., Müller, P., Zhang, E.: Interactive procedural street modeling. ACM Trans. Graph. 27(3), 103:1–103:10 (2008). https://doi.org/10.1145/1360612.1360702

  9. Cordonnier, G., Braun, J., Cani, M.P., Benes, B., Galin, E., Peytavie, A., Guérin, E.: Large scale terrain generation from tectonic uplift and fluvial erosion. Comput. Graph. Forum 35(2), 165–175 (2016). https://doi.org/10.1111/cgf.12820

  10. Crandall, D.J., Backstrom, L., Huttenlocher, D., Kleinberg, J.: Mapping the world’s photos. In: Proceedings of the 18th International Conference on World Wide Web, pp. 761–770. ACM (2009)

  11. Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.A.: What makes Paris look like Paris? ACM Trans. Graph. 31(4), 101:1–101:9 (2012). https://doi.org/10.1145/2185520.2185597

  12. Fan, L., Musialski, P., Liu, L., Wonka, P.: Structure completion for facade layouts. ACM Trans. Graph. 33(6), 210:1–210:11 (2014). https://doi.org/10.1145/2661229.2661265

  13. Gain, J., Long, H., Cordonnier, G., Cani, M.P.:EcoBrush: interactive control of visually consistent large-scale ecosystems. Comput. Graph. Forum 36(2), 63–73 (2017). https://doi.org/10.1111/cgf.13107

  14. Galin, E., Peytavie, A., Guérin, E., Beneš, B.: Authoring hierarchical road networks. Comput. Graph. Forum 30(7), 2021–2030 (2011). https://doi.org/10.1111/j.1467-8659.2011.02055.x

  15. Galin, E., Peytavie, A., Maréchal, N., Guérin, E.: Procedural generation of roads. Comput. Graph. Forum 29(2), 429–438 (2010). https://doi.org/10.1111/j.1467-8659.2009.01612.x

    Article  Google Scholar 

  16. Génevaux, J.D., Galin, É., Guérin, E., Peytavie, A., Benes, B.: Terrain generation using procedural models based on hydrology. ACM Trans. Graph. (TOG) 32(4), 143 (2013)

    Article  MATH  Google Scholar 

  17. Génevaux, J.-D., Galin, E., Peytavie, A., Guérin, E., Briquet, C., Grosbellet, F., Benes, B.: Terrain modelling from feature primitives. Comput. Graph. Forum 34(6), 198–210 (2015). https://doi.org/10.1111/cgf.12530

    Article  Google Scholar 

  18. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)

  19. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

  20. Guérin, E., Digne, J., Galin, E., Peytavie, A.: Sparse representation of terrains for procedural modeling. Comput. Graph. Forum 35(2), 177–187 (2016). https://doi.org/10.1111/cgf.12821

    Article  Google Scholar 

  21. Guérin, E., Digne, J., Galin, E., Peytavie, A., Wolf, C., Benes, B., Martinez, B.: Interactive example-based terrain authoring with conditional generative adversarial networks. ACM Trans. Graph. (TOG) 36(6), 228 (2017)

    Article  Google Scholar 

  22. Guo, J., Cheng, Z., Xu, S., Zhang, X.: Realistic procedural plant modeling guided by 3D point cloud. In: ACM SIGGRAPH 2017 Posters, pp. 85:1–85:2. ACM, New York, NY, USA (2017). https://doi.org/10.1145/3102163.3102193

  23. Henricsson, O., Streilein, A., Gruen, A.: Automated 3-D reconstruction of buildings and visualization of city models (1996)

  24. Hou, F., Qin, H., Qi, Y.: Procedure-based component and architecture modeling from a single image. Vis. Comput. 32(2), 151–166 (2016)

    Article  Google Scholar 

  25. Huijser, R., Dobbe, J., Bronsvoort, W.F., Bidarra, R.: Procedural natural systems for game level design. In: 2010 Brazilian Symposium on Games and Digital Entertainment (SBGAMES), pp. 189–198. IEEE (2010)

  26. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 (2015)

  27. Kaňuk, J., Gallay, M., Hofierka, J.: Generating time series of virtual 3-D city models using a retrospective approach. Landsc. Urban Plan. 139, 40–53 (2015)

    Article  Google Scholar 

  28. Kelly, T., Femiani, J., Wonka, P., Mitra, N.J.: BigSUR: large-scale structured urban reconstruction. ACM Trans. Graph. 36(6), 204:1–204:16 (2017). https://doi.org/10.1145/3130800.3130823

  29. Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. arXiv:1412.6980 (2014)

  30. Kolbe, T.H., Gröger, G., CityGML: interoperable access to 3D city models. In: van Oosterom, P., Zlatanova, S., Fendel, E.M. (eds.) Geo-Information for Disaster Management. Springer, Berlin, Heidelberg (2005)

  31. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Advances Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386

    Article  Google Scholar 

  32. Lipp, M., Scherzer, D., Wonka, P., Wimmer, M.: Interactive modeling of city layouts using layers of procedural content. Comput. Graph. Forum 30(2), 345–354 (2011). https://doi.org/10.1111/j.1467-8659.2011.01865.x

    Article  Google Scholar 

  33. Lyu, X., Han, Q., de Vries, B.: Procedural modeling of urban layout: population, land use, and road network. Transp. Res. Procedia 25, 3333–3342 (2017)

    Article  Google Scholar 

  34. Ma, R., Li, H., Zou, C., Liao, Z., Tong, X., Zhang, H.: Action-driven 3D indoor scene evolution. ACM Trans. Graph. 35(6), 173-1 (2016)

    Article  Google Scholar 

  35. Metz, L., Poole, B., Pfau, D., Sohl-Dickstein, J.: Unrolled Generative Adversarial Networks. arXiv:1611.02163 (2016)

  36. Mirza, M., Osindero, S.: Conditional Generative Adversarial Nets. arXiv:1411.1784 (2014)

  37. Müller, P., Zeng, G., Wonka, P., Van Gool, L.: Image-based procedural modeling of facades. ACM Trans. Graph. (TOG) 26(3), 85 (2007)

    Article  Google Scholar 

  38. Nishida, G., Bousseau, A., Aliaga, D.G.: Procedural modeling of a building from a single image. Comput. Graph. Forum 37(2), 415–429 (2018). https://doi.org/10.1111/cgf.13372

    Article  Google Scholar 

  39. Nishida, G., Garcia-Dorado, I., Aliaga, D.G., Benes, B., Bousseau, A.: Interactive sketching of urban procedural models. ACM Trans. Graph. (TOG) 35(4), 130 (2016)

    Article  Google Scholar 

  40. Parish, Y.I., Müller, P.: Procedural modeling of cities. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 301–308. ACM (2001)

  41. Radford, A., Metz, L., Chintala, S.: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv:1511.06434 (2015)

  42. Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative adversarial text-to-image synthesis. In: Proceedings of The 33rd International Conference on Machine Learning (2016)

  43. Sharma, R.: Procedural city generator. In: International Conference System Modeling & Advancement in Research Trends (SMART), pp. 213–217. IEEE (2016)

  44. Sinha, S.N., Steedly, D., Szeliski, R., Agrawala, M., Pollefeys, M.: Interactive 3D architectural modeling from unordered photo collections. ACM Trans. Graph. 27(5), 159:1–159:10 (2008). https://doi.org/10.1145/1409060.1409112

  45. Smelik, R., Galka, K., De Kraker, K.J., Kuijper, F., Bidarra, R.: Semantic constraints for procedural generation of virtual worlds. In: Proceedings of the 2nd International Workshop on Procedural Content Generation in Games, p. 9. ACM (2011)

  46. Smelik, R.M., Tutenel, T., Bidarra, R., Benes, B.: A survey on procedural modelling for virtual worlds. Comput. Graph. Forum 33(6), 31–50 (2014). https://doi.org/10.1111/cgf.12276

    Article  Google Scholar 

  47. Smelik, R.M., Tutenel, T., de Kraker, K.J., Bidarra, R.: Interactive creation of virtual worlds using procedural sketching. In: Lensch, H.P.A., Seipel, S. (eds.) Eurographics 2010 - Short Papers, Norrköping, Sweden, May 3–7, 2010. Eurographics Association (2010)

  48. Smelik, R.M., Tutenel, T., de Kraker, K.J., Bidarra, R.: A declarative approach to procedural modeling of virtual worlds. Comput. Graph. 35(2), 352–363 (2011)

    Article  Google Scholar 

  49. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. 25(3), 835–846 (2006). https://doi.org/10.1145/1141911.1141964

    Article  Google Scholar 

  50. Snavely, N., Seitz, S.M., Szeliski, R.: Modeling the world from internet photo collections. Int. J. Comput. Vis. 80(2), 189–210 (2008)

    Article  Google Scholar 

  51. Sun, J., Yu, X., Baciu, G., Green, M.: Template-based generation of road networks for virtual city modeling. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology, pp. 33–40. ACM (2002)

  52. Talton, J.O., Lou, Y., Lesser, S., Duke, J., Měch, R., Koltun, V.: Metropolis procedural modeling. ACM Trans. Graph. (TOG) 30(2), 11 (2011)

    Article  Google Scholar 

  53. Vanegas, C.A., Garcia-Dorado, I., Aliaga, D.G., Benes, B., Waddell, P.: Inverse design of urban procedural models. ACM Trans. Graph. (TOG) 31(6), 168 (2012)

    Article  Google Scholar 

  54. Vanegas, C.A., Kelly, T., Weber, B., Halatsch, J., Aliaga, D.G., Müller, P.: Procedural generation of parcels in urban modeling. Comput. Graph. Forum 31(2pt3), 681–690 (2012). https://doi.org/10.1111/j.1467-8659.2012.03047.x

  55. Vanek, J., Benes, B., Herout, A., Stava, O.: Large-scale physics-based terrain editing using adaptive tiles on the GPU. IEEE Comput. Graph. Appl. 31(6), 35–44 (2011)

    Article  Google Scholar 

  56. Vezhnevets, V., Konushin, A., Ignatenko, A.: Interactive image-based urban modeling. In: Proceedings of PIA, pp. 63–68 (2007)

  57. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  58. Wolberg, G., Zokai, S.: PhotoSketch: a photocentric urban 3D modeling system. Vis. Comput. 34(5), 605–616 (2018). https://doi.org/10.1007/s00371-017-1365-x

    Article  Google Scholar 

  59. Wonka, P., Aliaga, D., Müller, P., Vanegas, C.: Modeling 3D urban spaces using procedural and simulation-based techniques. In: ACM SIGGRAPH 2011 Courses, p. 9. ACM (2011)

  60. Yang, Y.L., Wang, J., Vouga, E., Wonka, P.: Urban pattern: layout design by hierarchical domain splitting. ACM Trans. Graph. (TOG) 32(6), 181 (2013)

    Article  Google Scholar 

  61. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: Proceedings of CVPR (2017)

Download references

Funding

This study was funded by National Research Foundation of Korea (NRF) (Grant Number 2015R1D1A1A09060399).

Author information

Authors and Affiliations

  1. School of Computing, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea

    Suzi Kim & Sunghee Choi

  2. Samsung Research of Samsung Electronics Co., Ltd., Suwon, South Korea

    Dodam Kim

Authors
  1. Suzi Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dodam Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sunghee Choi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sunghee Choi.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, S., Kim, D. & Choi, S. CityCraft: 3D virtual city creation from a single image. Vis Comput 36, 911–924 (2020). https://doi.org/10.1007/s00371-019-01701-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01701-x

Keywords

Search

Navigation