Skip to main content

A Stratified Pipeline for Vehicle Inpainting in Orthophotos

  • Conference paper
  • First Online:
Pattern Recognition (ICPR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15322))

Included in the following conference series:

  • 267 Accesses

  • 1 Citation

Abstract

The paper outlines a pipeline for the removal of transient objects from orthophotos to enhance the clarity and utility for orthophotos in military and civilian geo-databases generation as the main application. The presented deep-learning-based pipeline includes detecting the objects of interest, masking them out, and using the image and an enhanced inpainting mask to fill in these areas seamlessly. The approach combines semantic segmentation, utilizing an adapted DeepLabv3+ model, with shadow detection using Particle Swarm Optimization, and concludes with a generative inpainting process using a three-stage Generative Adversarial Network (3GAN) system for edge, segmentation, and texture inpainting. This method is applied to a well-known remote sensing dataset for detailed analysis, highlighting the integrated approach’s effectiveness in creating realistic, cleaned-up orthophotos.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://time.com/6266606/how-to-spot-deepfake-pope/

References

  1. Ammour, N., Alhichri, H., Bazi, Y., Benjdira, B., Alajlan, N., Zuair, M.: Deep learning approach for car detection in UAV imagery. Remote Sensing 9(4/312), 1–15 (2017)

    Google Scholar 

  2. Bulatov, D., Häufel, G., Meidow, J., Pohl, M., Solbrig, P., Wernerus, P.: Context-based automatic reconstruction and texturing of 3D urban terrain for quick-response tasks. ISPRS J. Photogramm. Remote. Sens. 93, 157–170 (2014)

    Article  Google Scholar 

  3. Chen, H., Luo, Y., Cao, L., Zhang, B., Guo, G., Wang, C., Li, J., Ji, R.: Generalized zero-shot vehicle detection in remote sensing imagery via coarse-to-fine framework. In: International Joint Conference on Artificial Intelligence. pp. 687–693 (2019)

    Google Scholar 

  4. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)

    Article  Google Scholar 

  5. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: European Conference on Computer Vision. pp. 801–818 (2018)

    Google Scholar 

  6. Chen, X., Xiang, S., Liu, C.L., Pan, C.H.: Vehicle detection in satellite images by hybrid deep convolutional neural networks. IEEE Geosci. Remote Sens. Lett. 11(10), 1797–1801 (2014)

    Article  Google Scholar 

  7. Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)

    Article  Google Scholar 

  8. Elharrouss, O., Almaadeed, N., Al-Maadeed, S., Akbari, Y.: Image inpainting: A review. Neural Process. Lett. 51, 2007–2028 (2019)

    Article  Google Scholar 

  9. Frommholz, D., Kuijper, F., Bulatov, D., Cheung, D.: Geospecific terrain databases for military simulation environments. In: Electro-Optical Remote Sensing XVI. vol. 12272, pp. 46–59. SPIE (2022)

    Google Scholar 

  10. Guo, S., Xiong, X., Liu, Z., Bai, X., Zhou, F.: Infrared simulation of large-scale urban scene through LOD. Opt. Express 26(18), 23980–24002 (2018)

    Article  Google Scholar 

  11. He, Z., Zhang, Z., Guo, M., Wu, L., Huang, Y.: Adaptive unsupervised-shadow-detection approach for remote-sensing image based on multichannel features. Remote Sensing 14(12), 2756 (2022)

    Article  Google Scholar 

  12. Huang, Z., Qin, C., Liu, R., Weng, Z., Zhu, Y.: Semantic-aware context aggregation for image inpainting. In: Proc. IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 2465–2469. IEEE (2021)

    Google Scholar 

  13. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Transactions on Graphics (TOG) 36(4), 1–14 (2017)

    Article  Google Scholar 

  14. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1125–1134 (2017)

    Google Scholar 

  15. Ji, H., Gao, Z., Mei, T., Ramesh, B.: Vehicle detection in remote sensing images leveraging on simultaneous super-resolution. IEEE Geosci. Remote Sens. Lett. 17(4), 676–680 (2019)

    Article  Google Scholar 

  16. Kottler, B., List, L., Bulatov, D., Weinmann, M.: 3GAN: A three-gan-based approach for image inpainting applied to the reconstruction of occluded parts of building walls. In: VISIGRAPP (4: VISAPP). pp. 427–435 (2022)

    Google Scholar 

  17. Leberl, F., Bischof, H., Grabner, H., Kluckner, S.: Recognizing cars in aerial imagery to improve orthophotos. In: Proc. ACM International Symposium on Advances in Geographic Information Systems. p. 2. ACM (2007)

    Google Scholar 

  18. Li, J., Wang, N., Zhang, L., Du, B., Tao, D.: Recurrent feature reasoning for image inpainting. In: Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7760–7768 (2020)

    Google Scholar 

  19. Liao, L., Xiao, J., Wang, Z., Lin, C.W., Satoh, S.: Guidance and evaluation: Semantic-aware image inpainting for mixed scenes. In: Proc. 16th European Conference on Computer Vision, Part XXVII 16. pp. 683–700. Springer (2020)

    Google Scholar 

  20. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 85–100 (2018)

    Google Scholar 

  21. Liu, Y., Piramanayagam, S., Monteiro, S.T., Saber, E.: Dense semantic labeling of very-high-resolution aerial imagery and lidar with fully-convolutional neural networks and higher-order CRFs. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 76–85 (2017)

    Google Scholar 

  22. Mo, N., Yan, L.: Improved faster RCNN based on feature amplification and oversampling data augmentation for oriented vehicle detection in aerial images. Remote Sensing 12(16/2558), 1–21 (2020)

    Google Scholar 

  23. Nazeri, K., Ng, E., Joseph, T., Qureshi, F.Z., Ebrahimi, M.: EdgeConnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212 (2019)

  24. Park, J., Cho, Y.K., Kim, S.: Deep learning-based UAV image segmentation and inpainting for generating vehicle-free orthomosaic. Int. J. Appl. Earth Obs. Geoinf. 115, 103111 (2022)

    Google Scholar 

  25. Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition. pp. 2536–2544 (2016)

    Google Scholar 

  26. Pyo, J., Rocha, Y.G., Ghosh, A., Lee, K., In, G., Kuc, T.: Object removal and inpainting from image using combined GANs. In: Proc. 20th International Conference on Control, Automation and Systems (ICCAS). pp. 1116–1119 (2020)

    Google Scholar 

  27. Qiu, K., Bulatov, D., Lucks, L.: Improving car detection from aerial footage with elevation information and markov random fields. In: Proceedings of the 19th International Conference on Signal Processing and Multimedia Applications, SIGMAP 2022. pp. 112–119. SCITEPRESS (2022)

    Google Scholar 

  28. Rottensteiner, F., Sohn, G., Gerke, M., Wegner, J.D., Breitkopf, U., Jung, J.: Results of the ISPRS benchmark on urban object detection and 3D building reconstruction. ISPRS J. Photogramm. Remote. Sens. 93, 256–271 (2014)

    Article  Google Scholar 

  29. Schilling, H., Bulatov, D., Niessner, R., Middelmann, W., Soergel, U.: Detection of vehicles in multisensor data via multibranch convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 11(11), 4299–4316 (2018)

    Article  Google Scholar 

  30. Schlagenhauf, T., Xia, Y., Fleischer, J.: Context-based image segment labeling (cbisl). arXiv preprint arXiv:2011.00784 (2020)

  31. Shalunts, G., Haxhimusa, Y., Sablatnig, R.: Architectural style classification of building facade windows. In: International Symposium on Visual Computing. pp. 280–289. Springer (2011)

    Google Scholar 

  32. Shao, H., Wang, Y., Fu, Y., Yin, Z.: Generative image inpainting via edge structure and color aware fusion. Signal Processing: Image Communication 87–115929, 1–9 (2020)

    Google Scholar 

  33. Song, Y., Yang, C., Lin, Z., Liu, X., Huang, Q., Li, H., Kuo, C.C.J.: Contextual-based image inpainting: Infer, match, and translate. In: Proc. IEEE European Conference on Computer Vision (ECCV). pp. 3–19 (2018)

    Google Scholar 

  34. Song, Y., Yang, C., Shen, Y., Wang, P., Huang, Q., Kuo, C.C.J.: Spg-net: Segmentation prediction and guidance network for image inpainting. In: Proc. British Machine Vision Conference. vol. 97, pp. 1–14 (2018)

    Google Scholar 

  35. Tang, T., Zhou, S., Deng, Z., Zou, H., Lei, L.: Vehicle detection in aerial images based on region convolutional neural networks and hard negative example mining. Sensors 17(2), 336 (2017)

    Article  Google Scholar 

  36. Tayara, H., Soo, K.G., Chong, K.T.: Vehicle detection and counting in high-resolution aerial images using convolutional regression neural network. IEEE Access 6, 2220–2230 (2017)

    Article  Google Scholar 

  37. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition. pp. 6721–6729 (2017)

    Google Scholar 

  38. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proc. IEEE/CVF International Conference on Computer Vision. pp. 4471–4480 (2019)

    Google Scholar 

  39. Zhang, J., Fukuda, T., Yabuki, N.: Automatic object removal with obstructed façades completion using semantic segmentation and generative adversarial inpainting. IEEE Access 9, 117486–117495 (2021)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Benedikt Kottler .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kottler, B., Qiu, K., Häufel, G., Bulatov, D. (2025). A Stratified Pipeline for Vehicle Inpainting in Orthophotos. In: Antonacopoulos, A., Chaudhuri, S., Chellappa, R., Liu, CL., Bhattacharya, S., Pal, U. (eds) Pattern Recognition. ICPR 2024. Lecture Notes in Computer Science, vol 15322. Springer, Cham. https://doi.org/10.1007/978-3-031-78312-8_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-78312-8_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-78311-1

  • Online ISBN: 978-3-031-78312-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics