Skip to main content
SpringerLink
Log in

An efficient approach for image de-fencing based on conditional generative adversarial network

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Automated image de-fencing is an important area of computer vision that deals with the problem of virtually removing fence structures, if any, from images and produce aesthetically pleasing images without the fence structures. Unlike most of the previous de-fencing approaches that employ a two-stage process of fence mask detection followed by image inpainting, here we present a single-stage end-to-end conditional generative adversarial network-based de-fencing model that takes as input a fenced image and produces the corresponding de-fenced image in only 16 ms. The proposed network has been trained using an extensive dataset of fenced and ground-truth de-fenced image pairs by employing a combination of adversarial loss, L1 loss, perceptual loss, and estimated fence mask loss till convergence. The experimental results shows that our approach is capable of successfully handling images with even broken, irregular, and occluded fence structures. Qualitative and quantitative comparative study with previous de-fencing methods also show that our approach outperforms these existing techniques in terms of both response time and quality of de-fencing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://bit.ly/defencing-dataset.

References

  1. Liu, Y., Belkina, T., Hays, J., Lublinerman, R.: Image de-fencing. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  2. Farid, M.S., Mahmood, A., Grangetto, M.: Image de-fencing framework with hybrid inpainting algorithm. Signal Image Video Process. 10(7), 1193–1201 (2016)

    Article  Google Scholar 

  3. Kumar, V., Mukherjee, J., Mandal, S.K.D.: Image defencing via signal demixing. In: Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, pp. 1–8 (2016)

  4. Khalid, M., Yousaf, M.M., Murtaza, K., Sarwar, S.M.: Image de-fencing using histograms of oriented gradients. Signal Image Video Process. 12(6), 1173–1180 (2018)

    Article  Google Scholar 

  5. Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

  6. Gupta, D., Jain, S., Tripathi, U., Chattopadhyay, P., Wang, L.: A robust and efficient image de-fencing approach using conditional generative adversarial networks. Signal Image Video Process. 15(2), 297–305 (2021)

    Article  Google Scholar 

  7. Hays, J., Leordeanu, M., Efros, A.A., Liu, Y.: Discovering texture regularity as a higher-order correspondence problem. In: Proceedings of the European Conference on Computer Vision, pp. 522–535 (2006)

  8. Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)

    Article  Google Scholar 

  9. Park, M., Brocklehurst, K., Collins, R.T., Liu, Y.: Image de-fencing revisited. In: Proceedings of the Asian Conference on Computer Vision, pp. 422–434 (2010)

  10. Kumar, V., Mukherjee, J., Mandal, S.K.D.: Combinatorial exemplar-based image inpainting. In: Proceedings of the Workshop on Combinatorial Image Analysis, pp. 284–298 (2015)

  11. Kumar, V., Mukhopadhyay, J., Mandal, S.K.D.: Modified exemplar-based image inpainting via primal-dual optimization. In: Proceedings of the International Conference on Pattern Recognition and Machine Intelligence, pp. 116–125 (2015)

  12. Shang, W., Zhu, P., Ren, D.: Semi-supervised learning to remove fences from a single image. In: Proceedings of the Chinese Conference on Pattern Recognition and Computer Vision, pp. 79–90 (2020)

  13. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 9446–9454 (2018)

  14. Jonna, S., Nakka, K.K., Sahay, R.R.: My camera can see through fences: a deep learning approach for image de-fencing. In: Proceedings of the Asian Conference on Pattern Recognition, pp. 261–265 (2015)

  15. Jonna, S., Nakka, K.K., Sahay, R.R.: Deep learning based fence segmentation and removal from an image using a video sequence. In: Proceedings of the European Conference on Computer Vision, pp. 836–851. Springer (2016)

  16. Du, C., Kang, B., Xu, Z., Dai, J., Nguyen, T.: Accurate and efficient video de-fencing using convolutional neural networks and temporal information. In: Proceedings of the International Conference on Multimedia and Expo, pp. 1–6 (2018)

  17. Negi, C.S., Mandal, K., Sahay, R.R., Kankanhalli, M.S.: Super-resolution de-fencing: simultaneous fence removal and high-resolution image recovery using videos. In: Proceedings of the Conference on Multimedia and Expo Workshops, pp. 1–6 (2014)

  18. Jonna, S., Satapathy, S., Sahay, R.R.: Stereo image de-fencing using smartphones. In: Proceedings of the Conference on Acoustics, Speech and Signal Processing, pp. 1792–1796 (2017)

  19. Yadong, M., Liu, W., Yan, S.: Video de-fencing. IEEE Trans. Circuits Syst. Video Technol. 24(7), 1111–1121 (2013)

    Google Scholar 

  20. Yi, R., Wang, J., Tan, P.: Automatic fence segmentation in videos of dynamic scenes. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 705–713 (2016)

  21. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Springer (2015)

  22. Li, C., Wand, M.: Precomputed real-time texture synthesis with markovian generative adversarial networks. In: European conference on computer vision, pp. 702–716. Springer (2016)

  23. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Proceedings of the European Conference on Computer Vision, pp. 694–711 (2016)

  24. Wen, L., Li, X., Li, X., Gao, L.: A new transfer learning based on VGG-19 network for fault diagnosis. In: 23rd International Conference on Computer Supported Cooperative Work in Design, pp. 205–209. IEEE (2019)

  25. Bradley, D., Roth, G.: Adaptive thresholding using the integral image. J. Graph. Tools 12(2), 13–21 (2007)

    Article  Google Scholar 

  26. Chen, Q., Xu, J., Koltun, V.: Fast image processing with fully-convolutional networks (2017)

  27. Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks (2017)

  28. Dosselmann, R., Yang, X.D.: A comprehensive assessment of the structural similarity index. Signal Image Video Process. 5(1), 81–91 (2011)

  29. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

Download references

Acknowledgements

The authors would like to thank NVIDIA for supporting their research with a Titan Xp GPU, and all those who participated in the survey for computing the MOS.

Author information

Authors and Affiliations

  1. Pattern Recognition Laboratory, Department of Computer Science and Engineering, Indian Institute of Technology (BHU), Varanasi, 221005, India

    Utkarsh Mishra, Josephine Crystal R. Mathew & Pratik Chattopadhyay

  2. Department of Mathematical Sciences, Indian Institute of Technology (BHU), Varanasi, 221005, India

    Akshat Agrawal & Rajesh K. Pandey

Authors
  1. Utkarsh Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Akshat Agrawal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Josephine Crystal R. Mathew
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rajesh K. Pandey
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pratik Chattopadhyay
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pratik Chattopadhyay.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mishra, U., Agrawal, A., Mathew, J.C.R. et al. An efficient approach for image de-fencing based on conditional generative adversarial network. SIViP 17, 147–155 (2023). https://doi.org/10.1007/s11760-022-02215-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02215-1

Keywords

Search

Navigation