Skip to main content
SpringerLink
Log in

MRS-Net: an image inpainting algorithm with multi-scale residual attention fusion

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Image repair is to repair partially damaged images. At present, when repairing the images with arbitrary missing shape and the images with large defect area, the current methods have some problems, such as fuzzy image repair and differences in the repaired joints. Therefore, this paper proposes an image repair model of MRS-Net(Multiscale Residual Squeeze-and-congestion Networks) by using the generated countermeasure network to repair the images with arbitrary missing shape and large defect area. The generator adds a residual attention module in the connection layer between the encoder and the decoder to improve the repair ability of the model, and generates a network through multi-scale joint feedback against loss, reconstruction loss, perception loss, style loss and total variation loss, so as to ensure the visual consistency between the repair boundary and the surrounding real image. At the same time, the binary cross entropy loss feedback discriminant network is used. The proposed model is trained and tested on the dataset CelebA and Oxford buildings. Experiments show that the proposed model can effectively extract the missing information, Meanwhile, the repair results have natural transition boundaries and clear details. MRS-Net can improve SSIM(structural similarity) index by about 2% - 5% and PSNR(peak signal-to-noise ratio)index by about 1-3. The FID(Frechet Inception Distance score) straight index is reduced by 2-8, and the evaluation indexes are improved.when the missing area accounts for 5% - 10%, 11% - 20%, 21% - 30%, 31% - 40% and 41% - 50%,MRS-Net has better image restoration effect, which can reflect that MRS-Net has better robustness.The proposed image restoration method can repair faces, buildings and other scenes. At the same time, it can repair defects in different shapes and areas.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Wang X (2007) Application of image inpainting based on partial differential equation [D]. Anhui University, Hefei

    Google Scholar 

  2. Rudin LI, Osher S (1994) Total variation based image restoration with free local constraints[C]. Proceedings of 1st International Conference on Image Processing 1:31–35

    Article  Google Scholar 

  3. He K, Zhang T, Liang R (2010) An improved image inpainting algorithm based on total variation model [J]. Optoelectron-icslaser 11(12):1890–1893

    Google Scholar 

  4. Shao X, Liu Z, Song B (2004) An adaptive image inpainting method based on TV model [J]. Acta Sinica Sinica Sinica 9(2):113–117

    Google Scholar 

  5. Chan TF, Shen J (2001) Nontexture inpainting by curvature drive diffusion[J]. J Vis Commun Image Rep 12(4):436–449

    Article  Google Scholar 

  6. Zhang L (2017) CDD patching model based on partial differential equation [J]. Modern Econ Inform 22(12):426–432

    Google Scholar 

  7. Bertalmio M, Saptro G, Caselles V, et al. (2000) Image inpainting[C]. In: Proceeding of 27th Annual conference on computer graphics and interactive techniques, pp 417–424

  8. Barnes C, Shechtman E, Finkelstein A et al (2009) Patch-Match: a ran-domized correspondence algorithm for structural image edit-ing[J]. ACM Trans Graph 28(3):1–11

    Article  Google Scholar 

  9. Wang J, Tang G, Liu X et al (2020) Structure group sparse representation image inpainting algorithm based on criminisi [J]. Comput Technol Develop 30(3):24–29

    Google Scholar 

  10. Ren S, Tang X, Kang J (2013) Image inpainting algorithm based on texture and edge features [J]. J Comput Aided Des Graph 25(11):1682–1693

    Google Scholar 

  11. Li Z, He H, Yin Z et al (2013) Adaptive image inpainting algorithm based on block structure sparsity [J]. Acta electronica Sinica 20(3):549–554

    Google Scholar 

  12. Chang C, Yin L, Fang B (2012) An improved criminisi image inpainting algorithm [J]. Comput Applic Softw 29(9):240–242

    Google Scholar 

  13. Lei M, Wang C, Xue Y et al (2009) A new image inpainting method for sample block [J]. Optoelectron Laser 14(5):677–680

    Google Scholar 

  14. Jiao L, Wang W, Li B, Zhao Q (2019) Improved block matching restoration algorithm of Wutaishan mural [J]. J Comput Aided Des Graph 31(01):118–125

    Article  Google Scholar 

  15. Oord A, Kalchbrenner N, Kavukcuoglu K (2016) Pixel recurrent neural networks[J]. arXiv:1601.06759

  16. Feng L, Zhang L, Zhang X (2019) Image inpainting based on extended convolution [J]. Comput Applic 40(3):825–831

    Google Scholar 

  17. Zhang X, Cheng L, Bai S, Zhang F (2020) Face image restoration based on variational self encoder [J]. J Comput Aided Des Graph 32(03):401–409

    Google Scholar 

  18. Pathak D, Krähenbühl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: feature learning by inpainting. In: 2016 IEEE Conference on computer vision and pattern recognition (CVPR), pp 2536–2544

  19. Chen J (2019) Face image inpainting based on cascade generation counterwork [J]. J UESTC 48 (6):910–917

    MathSciNet  Google Scholar 

  20. Zheng C, Cham T-J, Cai J (2019) Pluralistic image completion [C]. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 1438– 1447

  21. Iizuka S, Simo Serra E, Ishikawa H (2017) Globally and locally consistent image completion [J]. ACM Trans Graph 36(4):107

    Article  Google Scholar 

  22. Ding Y, Miao L, Wang C (2019) Face image restoration based on generative adversary network [J]. J Dalian Univ National 21(5):458–462

    Google Scholar 

  23. Liu Y, Wang M, Zhu L (2020) Image inpainting method based on new generator structure [J]. Comput Syst Applic 29(1):158– 163

    Google Scholar 

  24. Guo Z, Chen Z, Yu T et al (2019) Progressive image inpainting with full-resolution residual network[C]. In: Proceedings of the 27th ACM international conference on multimedia, pp 2496–2504

  25. Demir U, Unal G (2018) Patch-based image inpainting with generative adversarial networks[J]. arXiv:1803.07422

  26. Shin YG, Sagong MC, Yeo YJ et al (2020) Pepsi++: fast and lightweight network for image inpainting[J]. IEEE Trans. Neural Netw Learn Syst 32(1):252–265

    Article  Google Scholar 

  27. Yang Y, Guo X, Ma J et al (2019) Lafin: generative landmark guided face inpainting[J]. arXiv:1911.11394

  28. Cao J, Zhang Z et al (2020) Application of enhanced consistency generation confrontation network in mural restoration [J]. J Comput Aided Des Graph 32(8):1315–1323

    Google Scholar 

  29. Zhang Q, Sun LJ, Wang W et al (2020) Restoration and evaluation of cultural relic image based on generative confrontation network [J]. Packaging Eng 41(17):237–243

    Google Scholar 

  30. Jian Y, Xu L, Zhao J (2021) Occluded face restoration based on generative countermeasure network [J]. Electron Des Eng 29(4):66–70

    Google Scholar 

  31. Yu J, Lin Z, Yang J et al (2018) Generative image inpainting with contextual attention [C]. In: 2018IEEE Conference on computer vision and pattern recognition (CVPR), pp 5505–5514

  32. Nazeri K, Ng E, Ty J et al (2019) EdgeConnect: generative image inpainting with adversarial edge learning[C]. In: 2019 IEEE Conference on computer vision and pattern recognition(CVPR). arXiv:1901.00212

  33. Liu H, Jiang B, Xiao Y, Yang C (2019) Coherent semantic attention for image inpainting [C]. In: 2019 IEEE Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pp 4170–4179

  34. Yan Z, Li X, Li M et al (2018) Shift-net: image inpainting via deep feature rearrangement[C]. In: Proceedings of the European conference on computer vision (ECCV), pp 1–17

  35. Wang N, Li J, Zhang L et al (2019) MUSICAL: multi-scale image contextual attention learning for inpainting[C]. In: IJCAI, pp 3748–3754

  36. Yang W, Wang M, Zhang L (2020) Semantic face image inpainting based on dense connection block u-net [J]. Comput Applic, 177–184

  37. Li K, Zhang W, Shao M et al (2020) Multiscale generative countermeasure network image restoration algorithm [J]. Comput Sci Explor 14(1):159–170

    Google Scholar 

  38. Xiao F, Liu Y, Xia M (2021) Generative countermeasure network image restoration combined with perceptual attention mechanism * [J]. J Xi’an Univer Technol 41(2):198–205

    Google Scholar 

  39. Li J, Wang N, Zhang L, Du B, Tao D (2020) Recurrent feature reasoning for image inpainting. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition (CVPR), pp 7757–7765

  40. Chen Y, Zhang H, Liu L et al (2021) Research on image inpainting algorithm of improved GAN based on two-discriminations networks[J]. Appl Intell 51(6):3460–3474

    Article  Google Scholar 

  41. Goodfellow I, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial nets[J]. Advances in Neural Information Processing Systems, 27

  42. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein gan, arXiv:https://arxiv.org/abs/1701.07875

  43. Hu J, Shen L, Albanie S, Sun G, Wu E (2019) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42(8):2011–2023

    Article  Google Scholar 

  44. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution[C]. European conference on computer vision. Springer, Cham, pp 694–711

    Google Scholar 

  45. Yingnan S, Yao F, Ningjun Z (2021) A generative image inpainting network based on the attention transfer network across layer mechanism[J]. Optik, 167101

  46. Junyi L, Yao X, Li H (2020) Image style transfer based on perceptive adversarial network. J Hefei University of Technology: Natural Science 43(5):624–628

    Google Scholar 

  47. Wang YQ (2020) Research on unpaired heterogeneous face image generation algorithm based on generative adversarial network [D]. Nanjing University

  48. Weiyun P (2020) Research on image style transfer optimization method based on deep learning [D]. University of Electronic Science and Technology of China

  49. Haiyang L (2021) Research and application of image restoration technology based on generative adversation network [D]. Lanzhou Jiaotong University

  50. Wu K (2020) Synthesis and segmentation of multi-contrast in MRI of liver cancer based on generative adversation network [D]. Taiyuan University of Technology

  51. Zhenlong D, Haiyang S, Gome S et al (2019) . Optics and Precision Engineering 8. 21 (3):413–419

    Google Scholar 

Download references

Acknowledgements

This study was supported by research grants from the National Natural Science Foundation of China(61976150), Natural Science Foundation of Shanxi Province (201801D121135,201901D111091).

Author information

Authors and Affiliations

  1. College of Information and Computer, Taiyuan University of Technology, Taiyuan, China

    Hongxia Deng, Guanyu Qian, Dongsheng Luo, Xindong Lv & Haifang Li

  2. The School of Engineering, The University of Edinburgh, Edinburgh, UK

    Haoqi Liu

Authors
  1. Hongxia Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guanyu Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dongsheng Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xindong Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haoqi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Haifang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongxia Deng.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Deng, H., Qian, G., Luo, D. et al. MRS-Net: an image inpainting algorithm with multi-scale residual attention fusion. Appl Intell 53, 7497–7511 (2023). https://doi.org/10.1007/s10489-022-03866-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03866-w

Keywords

Search

Navigation