Skip to main content

Advertisement

Springer Nature Link
Log in

VResNet: A Deep Learning Architecture for Image Inpainting of Irregular Damaged Images

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

In computer vision, image inpainting is a famous problem to automatically reconstruct the damaged part of the image according to the undamaged portion of an image. Inpainting irregular damaged areas in the image is still challenging. Deep learning-based techniques have given us a fantastic performance over the last few years. In this paper, we propose VResNet, a deep-learning approach for image inpainting, inspired by U-Net architecture and the residual framework. Since deeper neural networks are extra hard to train, the superficial convolution block in U-Net architecture is replaced by the residual learning block in the proposed approach to simplify the training of deeper neural networks. To develop an effective and adaptable model, an extensive series of experiments was conducted using the Paris-Street-View dataset. Our proposed method achieved notable results, including a PSNR of 20.65, an SSIM of 0.65, an L1 Loss of 6.90, and a total loss (L\(_{\hbox {Total}}\)) of 0.30 on the Paris-Street-View dataset. These outcomes clearly demonstrate the superior performance of our model when compared to other techniques. The paper presents both qualitative and quantitative comparisons to provide a comprehensive assessment of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Availability of Data and Materials

The dataset used for research is available at link this link: https://github.com/pathak22/context-encoder/issues/24. The other materials and code for the paper can be requested from the corresponding author at this email Id: sariva03@gmail.com

References

  1. Ballester C, Bertalmio M, Caselles V, Sapiro G, Verdera J. Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans Image Process. 2001;10(8):1200–11.

    Article  MathSciNet  Google Scholar 

  2. Barnes C, Shechtman E, Finkelstein A, Goldman DB. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans Graph. 2009;28(3):24.

    Article  Google Scholar 

  3. Bertalmio M, Sapiro G, Caselles V, Ballester C. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000;417–424.

  4. Bishop CM et al. Neural networks for pattern recognition. Oxford university press, 1995.

  5. Chen B, Qi X, Zhou Y, Yang G, Zheng Y, Xiao B. Image splicing localization using residual image and residual-based fully convolutional network. J Vis Commun Image Represent. 2020;73: 102967.

    Article  Google Scholar 

  6. El Biach FZ, Iala I, Laanaya H, Minaoui K. Encoder-decoder based convolutional neural networks for image forgery detection. Multimedia Tools and Applications, 2021;1–18.

  7. Haritha L, Prajith CA. Image inpainting using deep learning techniques: A review. In 2023 International Conference on Control, Communication and Computing (ICCC), 2023;1–6.

  8. Hays J, Efros AA. Scene completion using millions of photographs. Commun ACM. 2008;51(10):87–94.

    Article  Google Scholar 

  9. He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, 2015;1026–1034.

  10. He K, Zhang X, Ren S, Sun J. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell. 2015;37(9):1904–16.

    Article  Google Scholar 

  11. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016;770–778.

  12. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017;4700–4708.

  13. Iizuka S, Simo-Serra E, Ishikawa H. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG). 2017;36(4):1–14.

    Article  Google Scholar 

  14. Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, 2015;448–456. PMLR.

  15. Janocha K, Czarnecki WM. On loss functions for deep neural networks in classification. arXiv preprint arXiv:1702.05659, 2017.

  16. Johnson J, Alahi A, Fei-Fei L. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, 2016;694–711. Springer.

  17. Kaur A, Raj A, Jayanthi N, Indu S. Inpainting of irregular holes in a manuscript using unet and partial convolution. In 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA), 2020;778–784. IEEE.

  18. Kim S-M, Sohn J-M, Kim D-S. A method for concrete crack detection using u-net based image inpainting technique. Journal of the Korea Society of Computer and Information. 2020;25(10):35–42.

    Article  Google Scholar 

  19. Kingma DP, Ba J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

  20. Kwatra V, Essa I, Bobick A, Kwatra N. Texture optimization for example-based synthesis. In ACM SIGGRAPH 2005 Papers, 2005;795–802.

  21. Liao L, Hu R, Xiao J, Wang Z. Artist-net: Decorating the inferred content with unified style for image inpainting. IEEE Access. 2019;7:36921–33.

    Article  Google Scholar 

  22. Liu G, Reda FA, Shih KJ, Wang T-C, Tao A, Catanzaro B. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV), 2018;85–100.

  23. Liu L, Liu Y. Load image inpainting: An improved u-net based load missing data recovery method. Appl Energy. 2022;327: 119988.

    Article  Google Scholar 

  24. Oord Avd, Kalchbrenner N, Vinyals O, Espeholt L, Graves A, Kavukcuoglu K. Conditional image generation with pixelcnn decoders. arXiv preprint arXiv:1606.05328, 2016.

  25. O’Shea K, Nash R. An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458, 2015.

  26. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016;2536–2544.

  27. Qiang Z, He L, Dai F, Zhang Q, Li J. Image inpainting based on improved deep convolutional auto-encoder network. Chin J Electron. 2020;29(6):1074–84.

    Article  Google Scholar 

  28. Qin Z, Zeng Q, Zong Y, Xu F. Image inpainting based on deep learning: A review. Displays. 2021;69: 102028.

    Article  Google Scholar 

  29. Qureshi MA, Deriche M, Beghdadi A, Amin A. A critical survey of state-of-the-art image inpainting quality assessment metrics. J Vis Commun Image Represent. 2017;49:177–91.

    Article  Google Scholar 

  30. Rafi M, Mukhopadhyay S. Image quilting for texture synthesis of grayscale images using gray-level co-occurrence matrix and restricted cross-correlation. In Progress in Advanced Computing and Intelligent Engineering, 2019;37–47. Springer.

  31. Ren JS, Xu L, Yan Q, Sun W. Shepard convolutional neural networks. Adv Neural Inf Process Syst. 2015;28:901–9.

    Google Scholar 

  32. Ripley BD. Pattern recognition and neural networks. Cambridge university press, 2007.

  33. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, 2015;234–241. Springer.

  34. Shen L, Hong R, Zhang H, Zhang H, Wang M. Single-shot semantic image inpainting with densely connected generative networks. In Proceedings of the 27th ACM International Conference on Multimedia, 2019:1861–1869.

  35. Simakov D, Caspi Y, Shechtman E, Irani M. Summarizing visual data using bidirectional similarity. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008;1–8. IEEE.

  36. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.

  37. Song Y, Yang C, Lin Z, Liu X, Huang Q, Li H, Kuo C-CJ. Contextual-based image inpainting: Infer, match, and translate. In Proceedings of the European Conference on Computer Vision (ECCV), 2018;3–19.

  38. Sun L, Jiang B, Yang C, Dai J, Zeng W. Repgan: image inpainting via residual partial connection and mask discriminator. International Journal of Machine Learning and Cybernetics, 2023;1–11.

  39. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015;1–9.

  40. Telea A. An image inpainting technique based on the fast marching method. Journal of graphics tools. 2004;9(1):23–34.

    Article  Google Scholar 

  41. Turaga DS, Chen Y, Caviedes J. No reference psnr estimation for compressed pictures. Signal Processing: Image Communication. 2004;19(2):173–84.

    Google Scholar 

  42. Uhrig J, Schneider N, Schneider L, Franke U, Brox T, Geiger A. Sparsity invariant cnns. In 2017 international conference on 3D Vision (3DV), 2017;11–20. IEEE.

  43. Wang N, Wang W, Hu W, Fenster A, Li S. Thanka mural inpainting based on multi-scale adaptive partial convolution and stroke-like mask. IEEE Trans Image Process. 2021;30:3720–33.

    Article  Google Scholar 

  44. Wang Q, Ma Y, Zhao K, Tian Y. A comprehensive survey of loss functions in machine learning. Annals of Data Science, 2020;1–26.

  45. Xiang H, Zou Q, Nawaz MA, Huang X, Zhang F, Yu H. Deep learning for image inpainting: A survey. Pattern Recogn. 2023;134: 109046.

    Article  Google Scholar 

  46. Xiang H, Zou Q, Nawaz MA, Huang X, Zhang F, Yu H. Deep learning for image inpainting: A survey. Pattern Recogn. 2023;134: 109046.

    Article  Google Scholar 

  47. Yang C, Lu X, Lin Z, Shechtman E, Wang O, Li H. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017;6721–6729.

  48. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018;5505–5514.

  49. Zamir A, Shah M. Image geo-localization based on multiple nearest neighbor feature matching using generalized graphs, 2014.

  50. Zhang X, Zhai D, Li T, Zhou Y, Lin Y. Image inpainting based on deep learning: A review. Information Fusion, 2022.

  51. Zhou C, Gu Z, Gao Y, Wang J. An improved style transfer algorithm using feedforward neural network for real-time image conversion. Sustainability. 2019;11(20):5673.

    Article  Google Scholar 

Download references

Funding

Not applicable.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Dr. B. R. Ambedkar National Institute of Technology, Jalandhar, 144011, India

    Sariva Sharma & Rajneesh Rani

Authors
  1. Sariva Sharma
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Rajneesh Rani
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

SS contributed to the research planning, implementation, analysis, and writing of the original draft. RR contributed to research planning, writing, and evaluation of the original draft.

Corresponding author

Correspondence to Sariva Sharma.

Ethics declarations

Conflict of Interest

There were no known competing financial interests or personal ties that could have influenced the research presented in this study.

Code Availability

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, S., Rani, R. VResNet: A Deep Learning Architecture for Image Inpainting of Irregular Damaged Images. SN COMPUT. SCI. 5, 209 (2024). https://doi.org/10.1007/s42979-023-02523-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-023-02523-4

Keywords