skip to main content
10.1145/3503161.3548297acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Unsupervised Textured Terrain Generation via Differentiable Rendering

Published: 10 October 2022 Publication History

Abstract

Constructing large-scale realistic terrains using modern modeling tools is an extremely challenging task even for professional users, undermining the effectiveness of video games, virtual reality, and other applications. In this paper, we present a step towards unsupervised and realistic modeling of textured terrains from DEM and satellite imagery, built upon two-stage illumination and texture optimization via differentiable rendering. First, a differentiable renderer for satellite imagery is established based on the Lambert diffuse model that allows inverse optimization of material and lighting parameters towards specific objective. Second, the original illumination direction of satellite imagery is recovered by reducing the difference between the shadow distribution generated by the renderer and that of the satellite image in YCrCb colour space, leveraging the abundant geometric information of DEM. Third, we propose to generate the original texture of the shadowed region by introducing visual consistency and smoothness constraints via differentiable rendering to arrive at an end-to-end unsupervised architecture. Comprehensive experiments demonstrate the effectiveness and efficiency of our proposed method as a potential tool to achieve virtual terrain modeling for widespread graphics applications.

Supplementary Material

MP4 File (MM22-fp2364.mp4)
Presentation video of Unsupervised Textured Terrain Generation via Differentiable Rendering.

References

[1]
Zhihao Chen, Lei Zhu, Liang Wan, Song Wang, Wei Feng, and Pheng-Ann Heng. 2020. A multi-task mean teacher for semi-supervised shadow detection. In Computer Vision and Pattern Recognition. 5611--5620.
[2]
Bin Ding, Chengjiang Long, Ling Zhang, and Chunxia Xiao. 2019. Argan: Attentive recurrent generative adversarial network for shadow detection and removal. In International Conference on Computer Vision. 10213--10222.
[3]
Guoshuai Dong, Weimin Huang, William AP Smith, and Peng Ren. 2020. A shadow constrained conditional generative adversarial net for SRTM data restoration. Remote Sensing of Environment, Vol. 237 (2020), 111602.
[4]
Tom G Farr, Paul A Rosen, Edward Caro, Robert Crippen, Riley Duren, Scott Hensley, Michael Kobrick, Mimi Paller, Ernesto Rodriguez, Ladislav Roth, et al. 2007. The shuttle radar topography mission. Reviews of geophysics, Vol. 45, 2 (2007).
[5]
Anna Frü hstü ck, Ibraheem Alhashim, and Peter Wonka. 2019. TileGAN: synthesis of large-scale non-homogeneous textures. ACM Transactions on Graphics, Vol. 38, 4 (2019), 58:1--58:11.
[6]
Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou. 2019. Ganfit: Generative adversarial network fitting for high fidelity 3d face reconstruction. In Computer Vision and Pattern Recognition. 1155--1164.
[7]
Han Gong and Darren Cosker. 2014. Interactive Shadow Removal and Ground Truth for Variable Scene Categories. In British Machine Vision Conference. Citeseer, 1--11.
[8]
Yingqing He, Yazhou Xing, Tianjia Zhang, and Qifeng Chen. 2021. Unsupervised Portrait Shadow Removal via Generative Priors. In International Conference on Multimedia. 236--244.
[9]
James Hogan and William AP Smith. 2010. Refinement of digital elevation models from shadowing cues. In Computer Vision and Pattern Recognition. IEEE, 1181--1188.
[10]
Xiaowei Hu, Yitong Jiang, Chi-Wing Fu, and Pheng-Ann Heng. 2019. Mask-ShadowGAN: Learning to remove shadows from unpaired data. In International Conference on Computer Vision. 2472--2481.
[11]
Xiaowei Hu, Lei Zhu, Chi-Wing Fu, Jing Qin, and Pheng-Ann Heng. 2018. Direction-aware spatial context features for shadow detection. In Computer Vision and Pattern Recognition. 7454--7462.
[12]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Computer Vision and Pattern Recognition. 1125--1134.
[13]
Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. 2018. Learning category-specific mesh reconstruction from image collections. In European Conference on Computer Vision. 371--386.
[14]
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020. Analyzing and improving the image quality of stylegan. In Computer Vision and Pattern Recognition. 8110--8119.
[15]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations. 1--15.
[16]
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular Primitives for High-Performance Differentiable Rendering. ACM Transactions on Graphics, Vol. 39, 6 (2020).
[17]
Hieu Le and Dimitris Samaras. 2019. Shadow removal via shadow image decomposition. In International Conference on Computer Vision. 8578--8587.
[18]
Hieu Le and Dimitris Samaras. 2020. From shadow segmentation to shadow removal. In European Conference on Computer Vision. Springer, 264--281.
[19]
Gun-Hee Lee and Seong-Whan Lee. 2020. Uncertainty-aware mesh decoder for high fidelity 3d face reconstruction. In Computer Vision and Pattern Recognition. 6100--6109.
[20]
Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz. 2020. Self-supervised single-view 3d reconstruction via semantic consistency. In European Conference on Computer Vision. Springer, 677--693.
[21]
Yun-Hsuan Lin, Wen-Chin Chen, and Yung-Yu Chuang. 2020. Bedsr-net: A deep shadow removal network from a single document image. In Computer Vision and Pattern Recognition. 12905--12914.
[22]
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image inpainting for irregular holes using partial convolutions. In European Conference on Computer Vision. 85--100.
[23]
Zhihao Liu, Hui Yin, Yang Mi, Mengyang Pu, and Song Wang. 2021. Shadow removal by a lightness-guided network with training on unpaired data. IEEE Transactions on Image Processing, Vol. 30 (2021), 1853--1865.
[24]
Saritha Murali, VK Govindan, and Saidalavi Kalady. 2021. Quaternion-based image shadow removal. The Visual Computer (2021), 1--12.
[25]
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi. 2019. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212 (2019).
[26]
Vu Nguyen, Tomas F Yago Vicente, Maozheng Zhao, Minh Hoai, and Dimitris Samaras. 2017. Shadow detection with conditional generative adversarial networks. In International Conference on Computer Vision. 4510--4518.
[27]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Conference on Neural Information Processing Systems, Vol. 32. 8026--8037.
[28]
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Computer Vision and Pattern Recognition. 2536--2544.
[29]
Liangqiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson WH Lau. 2017. Deshadownet: A multi-context embedding deep network for shadow removal. In Computer Vision and Pattern Recognition. 4067--4075.
[30]
Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. 2019. Structureflow: Image inpainting via structure-aware appearance flow. In International Conference on Computer Vision. 181--190.
[31]
Ernesto Rodriguez, CS Morris, JE Belz, EC Chapin, JM Martin, W Daffer, and S Hensley. 2005. An assessment of the SRTM topographic products. (2005).
[32]
Ruben M Smelik, Tim Tutenel, Rafael Bidarra, and Bedrich Benes. 2014. A survey on procedural modelling for virtual worlds., Vol. 33, 6 (2014), 31--50.
[33]
PM Teillet, B Guindon, and DG Goodenough. 1982. On the slope-aspect correction of multispectral scanner data. Canadian Journal of Remote Sensing, Vol. 8, 2 (1982), 84--106.
[34]
Vasilis Toulatzis and Ioannis Fudos. 2019. Deep Terrain Expansion: Terrain Texture Synthesis with Deep Learning. In Computer Graphics & Visual Computing. 95--96.
[35]
Jifeng Wang, Xiang Li, and Jian Yang. 2018. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In Computer Vision and Pattern Recognition. 1788--1797.
[36]
Anna Wendleder, Andreas Felbier, Birgit Wessel, Martin Huber, and Achim Roth. 2016. A method to estimate long-wave height errors of SRTM C-band DEM. Geoscience and Remote Sensing Letters, Vol. 13, 5 (2016), 696--700.
[37]
Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding. 2019. Image inpainting with learnable bidirectional attention maps. In International Conference on Computer Vision. 8858--8867.
[38]
Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. 2017. High-resolution image inpainting using multi-scale neural patch synthesis. In Computer Vision and Pattern Recognition. 6721--6729.
[39]
Jie Yang, Zhiquan Qi, and Yong Shi. 2020. Learning to incorporate structure knowledge for image inpainting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12605--12612.
[40]
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2019. Free-form image inpainting with gated convolution. In International Conference on Computer Vision. 4471--4480.
[41]
Xiaoming Yu, Ge Li, Zhenqiang Ying, and Xiaoqiang Guo. 2017. A new shadow removal method using color-lines. In International Conference on Computer Analysis of Images and Patterns. Springer, 307--319.
[42]
Yu Zeng, Zhe Lin, Huchuan Lu, and Vishal M Patel. 2021. Cr-fill: Generative image inpainting with auxiliary contextual reconstruction. In International Conference on Computer Vision. 14164--14173.
[43]
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. 2021. Tfill: Image completion via a transformer-based architecture. arXiv preprint arXiv:2104.00845 (2021).
[44]
Silvia Zuffi, Angjoo Kanazawa, Tanya Berger-Wolf, and Michael J Black. 2019. Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images" In the Wild". In International Conference on Computer Vision. 5359--5368.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MM '22: Proceedings of the 30th ACM International Conference on Multimedia
October 2022
7537 pages
ISBN:9781450392037
DOI:10.1145/3503161
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. differentiable rendering
  2. generative model
  3. terrain texture

Qualifiers

  • Research-article

Funding Sources

  • National Science Foundation of USA
  • Natural Science Foundation of China

Conference

MM '22
Sponsor:

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 190
    Total Downloads
  • Downloads (Last 12 months)40
  • Downloads (Last 6 weeks)1
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media