Abstract
Many municipalities provide textured 3D city models for planning and simulation purposes. Usually, the textures are automatically taken from oblique aerial images. In these images, walls may be occluded by building parts, vegetation and other objects such as cars, traffic signs, etc. To obtain high quality models, these objects have to be segmented and then removed from facade textures. In this study, we investigate the ability of different non-specialized inpainting algorithms to continue facade patterns in occluded facade areas. To this end, non-occluded facade textures of a 3D city model are equipped with various masks simulating occlusions. Then, the performance of the algorithms is evaluated by comparing their results with the original images. In particular, very useful results are obtained with the neural network “DeepFill v2” trained with transfer learning on freely available facade datasets and the “Shift-Map” algorithm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
https://opencv.org (all websites accessed on January 12, 2022).
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
References
Bertalmio, M., Bertozzi, A., Shapiro, G.: Navier-Stokes, fluid dynamics, and image and video inpainting. In: Proceedings of CVPR 2001 (2001). https://doi.org/10.1109/CVPR.2001.990497
Cohen, A., Oswald, M.R., Liu, Y., Pollefeys, M.: Symmetry-aware façade parsing with occlusions. In: Proceedings of 2017 International Conference on 3D Vision (3DV), pp. 393–401 (2017). https://doi.org/10.1109/3DV.2017.00052
Dai, D., Riemenschneider, H., Schmitt, G., Van, L.: Example-based facade texture synthesis. In: Proceedings of 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1065–1072. IEEE Computer Society, Los Alamitos, CA (2013)
Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.: What makes Paris look like Paris? ACM Trans. Graph. 31(4:101), 1–9 (2012)
Genser, N., Seiler, J., Schilling, F., Kaup, A.: Signal and loss geometry aware frequency selective extrapolation for error concealment. In: Proceedings of 2018 Picture Coding Symposium (PCS), pp. 159–163 (2018)
Goebbels, S., Pohle-Fröhlich, R.: Roof reconstruction from airborne laser scanning data based on image processing methods. ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. III-3, 407–414 (2016)
Goebbels, S., Pohle-Fröhlich, R.: Automatic unfolding of CityGML buildings to paper models. Geographies 1(3), 333–345 (2021)
He, K., Sun, J.: Statistics of patch offsets for image completion. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7573, pp. 16–29. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33709-3_2
Hensel, S., Goebbels, S., Kada, M.: LSTM architectures for facade structure completion. In: Proceedings of GRAPP 2021, pp. 15–24 (2021)
Huang, J.B., Kang, S.B., Ahuja, N., Kopf, J.: Image completion using planar structure guidance. ACM Trans. Graph. 33(4), 1–10 (2014)
Kaup, A., Meisinger, K., Aach, T.: Frequency selective signal extrapolation with applications to error concealment in image communication. AEUE - Int. J. Electron. Commun. 59, 147–156 (2005)
Korah, T., Rasmussen, C.: Spatiotemporal inpainting for recovering texture maps of occluded building facades. IEEE Trans. Image Process. 16(9), 2262–2271 (2007)
Kottler, B., Bulatov, D., Zhang, X.: Context-aware patch-based method for façade inpainting. In: Proceedings of GRAPP 2020, pp. 210–218 (2020)
Kottler, B., List, L., Bulatov, D., Weinmann, M.: 3GAN: a three-GAN-based approach for image inpainting applied to the reconstruction of occluded parts of building walls. In: Proceedings of VISAPP 2022 (2022)
Mehra, S., Dogra, A., Goyal, B., Sharma, A.M., Chandra, R.: From textural inpainting to deep generative models: an extensive survey of image inpainting techniques. J. Comput. Sci. 16(1), 35–49 (2020)
Nazeri, K., Ng, E., Joseph, T., Qureshi, F.Z., Ebrahimi, M.: EdgeConnect: generative image inpainting with adversarial edge learning. arXiv Preprint: arXiv: 1901.00212 (2019)
Pritch, Y., Kav-Venaki, E., Peleg, S.: Shift-map image editing. In: Proceedings of 2009 IEEE International Conference on Computer Vision (ICCV), pp. 151–158. IEEE Computer Society, Los Alamitos, CA (2009)
Seiler, J., Jonscher, M., Schöberl, M., Kaup, A.: Resampling images to a regular grid from a non-regular subset of pixel positions using frequency selective reconstruction. IEEE Trans. Image Process. 24(11), 4540–4555 (2015)
Shen, H., et al.: Missing information reconstruction of remote sensing data: a technical review. IEEE Geosci. Remote Sens. Mag. 3(3), 61–85 (2015)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of 3rd International Conference on Learning Representations (ICLR) 2015, San Diego, CA, USA (2015)
Telea, A.: An image inpainting technique based on the fast marching method. J. Graph. Tools 9(1), 23–34 (2004)
Tyleček, R., Šára, R.: Spatial pattern templates for recognition of objects with regular structure. In: Weickert, J., Hein, M., Schiele, B. (eds.) GCPR 2013. LNCS, vol. 8142, pp. 364–374. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40602-7_39
Wang, Y., Tao, X., Qi, X., Shen, X., Jia, J.: Image inpainting via generative multi-column convolutional neural networks. In: Proceedings of 32nd International Conference on Neural Information Processing Systems, pp. 329–338. NIPS 2018. Curran Associates Inc., Red Hook, NY (2018)
Wonka, P., Wimmer, M., Sillion, F., Ribarsky, W.: Instant architecture. ACM Trans. Graph. 22(3), 669–677 (2003)
Yeh, Y.T., Breeden, K., Yang, L., Fisher, M., Hanrahan, P.: Synthesis of tiled patterns using factor graphs. ACM Trans. Graph. 32(1), 1–13 (2013)
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.: Generative image inpainting with contextual attention. In: Proceedings of CVPR 2018, arXiv preprint arXiv:1801.07892 (2018)
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.: Free-form image inpainting with gated convolution. In: Proceedings of CVPR 2019, arXiv preprint arXiv:1806.03589 (2019)
Zhang, J., Fukuda, T., Yabuki, N.: Automatic object removal with obstructed façades completion using semantic segmentation and generative adversarial inpainting. IEEE Access 9, 117486–117495 (2021)
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 40(6), 1452–14649 (2018)
Acknowledgements
This work was supported by a generous hardware grant from NVIDIA. The authors thank Udo Hannok from the cadastral office of the city of Krefeld for providing the oblique aerial images. The authors are also grateful to Regina Pohle-Fröhlich for valuable comments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Fritzsche, W., Goebbels, S., Hensel, S., Rußinski, M., Schuch, N. (2022). Inpainting Applied to Facade Images: A Comparison of Algorithms. In: El Yacoubi, M., Granger, E., Yuen, P.C., Pal, U., Vincent, N. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2022. Lecture Notes in Computer Science, vol 13363. Springer, Cham. https://doi.org/10.1007/978-3-031-09037-0_34
Download citation
DOI: https://doi.org/10.1007/978-3-031-09037-0_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-09036-3
Online ISBN: 978-3-031-09037-0
eBook Packages: Computer ScienceComputer Science (R0)