Skip to main content
SpringerLink
Log in

Multi-scale patch-GAN with edge detection for image inpainting

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Image inpainting with large missing blocks is quite challenging to obtain visual consistency and realistic effect. In this paper, the multi-scale patch generative adversarial networks with edge detection image inpainting (MPGE) was proposed. Firstly, an edge detector was introduced into the generator of multi-scale generative adversarial networks (GAN) to guide the inpainting of the edge contour in the image inpainting, which improved the inpainting effect of image posture and expression. Secondly, we designed a patch-GAN as the local discriminant to capture high frequency, and a function L2-loss was utilized to keep the high resolution and style of the original image. Thirdly, a multi-head attention mechanism was introduced into the generator and local discriminator to build a multilevel and multi-dimensional dependent network model for image subspaces, which improved the global consistency of the inpainted image. Finally, by finding the minimum data set with similar network expression ability, we quickly obtained the optimal value of multi-head. Thereby, a lot of training time was saved. The experiments conducted on Celeba dataset proved that our proposed algorithm quantitatively and qualitatively outperformed the baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Kim D, Woo S, Lee JY (2019) Recurrent temporal aggregation framework for deep video inpainting. Proc IEEE Trans Pattern Anal Mach Intell 42(5):1038–1052

    Article  Google Scholar 

  2. Li Z, Cai A, Wang L (2019) Promising generative adversarial network based sinogram inpainting method for ultra-limited-angle computed tomography imaging. Sensors 19(18):39–41

    Article  Google Scholar 

  3. Szeto R, Sun X, Lu K (2019) A temporally-aware interpolation network for video frame inpainting. IEEE Trans Pattern Anal Mach Intell 42(5):1053–1068

    Article  Google Scholar 

  4. Lander K, Bruce V, Bindemann M (2018) Use-inspired basic research on individual differences in face identification: implications for criminal investigation and security. Cogn Res: Princ Implications 3(1):1–13

    Article  Google Scholar 

  5. Zhang C, Li R, Chen W (2020) On the research of cultural relic restoration under reverse design. E3S Web Conf 189:03006

    Article  Google Scholar 

  6. Dalca AV, Bouman KL, Freeman WT (2018) Medical image imputation from image collections. IEEE Trans Med Imaging 38(2):504–514

    Article  Google Scholar 

  7. Iglesias FA, Feller A, Nagaraju K (2016) High-resolution, high-sensitivity, ground-based solar spectropolarimetry with a new fast imaging polarimeter-I. Prototyp Charact Astron Astrophys 590:A89

    Article  Google Scholar 

  8. Bertalmio M, Sapiro G, Caselles V (2000) Image Inpainting. In: Proceedings of Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, USA, 23–28, pp.417–424

  9. Criminisi A, Pérez P, Toyama K (2004) Region filling and object removal byexemplar-based image inpainting. IEEE Trans Image Process 13(9):1200–1212

    Article  Google Scholar 

  10. Bobin J, Starck JL, Fadili JM (2007) Morphological component analysis: an adaptive thresholding strategy. IEEE Trans Image Process 16(11):2675–2681

    Article  MATH  Google Scholar 

  11. Pathak D, Krahenbuhl P, Donahue J (n.d.) Context encoders: Feature learning by inpainting. Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas, USA, 26 June −01 July, pp: 2536–2544

  12. Iizuka S, Simo-Serra E, Ishikawa H (2017) Globally and locally consistent image completion. ACM Trans Graph (ToG) 36(4):1–14

    Article  Google Scholar 

  13. Demir U; Unal G (n.d.) Patch-based image inpainting with generative adversarial networks. 2018. Available online: https://Arxiv.Org/Abs/1803.07422. Access on 27 January 2021

  14. Park G, Han C, Yoon W (2020) MHSAN: Multi-Head Self-Attention Network for Visual Semantic Embedding. Snowmass Village, USA, 02–05, Mar; pp. 1518–1526

  15. Kim JH (2019) Estimating classification error rate: repeated cross-validation, repeated hold-out and bootstrap. Comput Stat Data Anal 53(11):3735–3745

    Article  MATH  Google Scholar 

  16. Richard M, Chang MKYS (2001). Fast digital image inpainting. Appeared in the Proceedings of the International Conference on Visualization, Imaging and Image Processing, Marbella, Spain, 3–5, September (2001) pp. 106–107

  17. Chan TF, Shen J (2001) Nontexture inpainting by curvature-driven diffusions. J Vis Commun Image Represent 12(4):436–449

    Article  Google Scholar 

  18. Shen J, Chan TF (2002) Mathematical models for local nontexture inpaintings. SIAM J Appl Math 62(3):1019–1043

    Article  MATH  Google Scholar 

  19. Chan TF, Shen J, Vese L (2003) Variational PDE models in image processing. Notices AMS 50(1):14–26

    MATH  Google Scholar 

  20. Xiang C, Cao Y, Duan P (2014) An improved exemplar-based image inpainting algorithm. 2014 9th International Conference on Computer Science & Education, Vancouver, BC, Canada , 22–24, pp. 770–775

  21. Aharon M, Elad M, Bruckstein A (2006) K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311–4322

    Article  MATH  Google Scholar 

  22. Abavisani M, Patel VM (2019) Deep sparse representation-based classification. IEEE Signal Process Lett 26(6):948–952

    Article  Google Scholar 

  23. Andreini P, Bonechi S, Bianchini M (2020) Image generation by GAN and style transfer for agar plate image segmentation. Comput Methods Prog Biomed 184:105268

    Article  Google Scholar 

  24. Izadyyazdanabadi M, Belykh E, Zhao X (2019) Fluorescence image histology pattern transformation using image style transfer. Front Oncol 9:519

    Article  Google Scholar 

  25. Savilahti EM, Haravuori H, Rytilä-Manninen M (2018) High Beck depression inventory 21 scores in adolescents without depression are associated with negative self-image and immature defense style. Psychiatry Res 263:61–68

    Article  Google Scholar 

  26. Xia B, Bao C (2014) Wiener filtering based speech enhancement with weighted denoising auto-encoder and noise classification. Speech Comm 60:13–29

    Article  Google Scholar 

  27. Yu J, Lin Z, Yang J (2018) Generative image inpainting with contextual attention. Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, USA, 18–23, pp. 5505–5514

  28. Wang Y, Tao X, Qi X (2018). Image inpainting via generative multi-column convolutional neural networks. vailable online: https://arxiv.org/abs/1810.08771. Access on 27 January 2021

  29. Zeng Y, Fu J, Chao H (2019) Learning pyramid-context encoder network for high-quality image inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 15–20, pp.1486–1494

  30. Zheng C, Cham TJ, Cai J (2019) Pluralistic image completion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 15–20, pp. 1438–1447

  31. Li J, Wang N, Zhang L (2020) Recurrent feature reasoning for image inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA,13–19, pp.7760–7768

  32. Chen Y, Zhang H, Liu L (2021) Research on image inpainting algorithm of improved GAN based on two-discriminations networks. Appl Intell 51(6):3460–3474

    Article  Google Scholar 

  33. Sagong M, Shin Y, Kim S (2019) PEPSI: Fast image inpainting with parallel decoding network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 15–20, June(2019); pp. 11360–11368

  34. Liu HY, Jiang B, Song Y (2020) Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, 23–28, August(2020); pp. 725–741

  35. Mnih V, Heess N, Graves A (2014) Recurrent models of visual attention. Advances in neural information processing systems, Montreal Convention palace, Canada, 8–13, pp. 2204–2212

  36. Xu K, Ba J, Kiros R (2015) Show, attend and tell: Neural image caption generation with visual attention. International conference on machine learning, Lille, France, 7–9, July (2015); pp. 2048–2057

  37. Zhang H, Goodfellow I, Metaxas D (2019) Self-attention generative adversarial networks. International conference on machine learning, London, U K, 08–10, pp. 7354–7363

  38. Cordonnier JB, Loukas A, Jaggi M (2019) On the relationship between self-attention and convolutional layers. Available online: https://arxiv.org/abs/1911.03584. Access on 27 January 2021

  39. Clark K, Khandelwal U, Levy O (2019) What does bert look at? an analysis of bert's attention. Available online: https://arxiv.org/abs/1906.04341. Access on 27 January 2021

  40. Correia GM, Niculae V, Martins AFT (2019) Adaptively sparse transformers. Available online: https://arxiv.org/abs/1909.00015. Access on 27 January 2021

  41. Xie S, Tu Z (2015) Holistically-nested edge detection. Proceedings of the IEEE international conference on computer vision, Santiago, Chile,13–16, December (2015);pp. 1395–1403

  42. Bertasius G, Shi J, Torresani L (2015) Deepedge: A multi-scale bifurcated deep network for top-down contour detection. Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA, 8–10, pp. 4380–4389

  43. Shen W, Wang X, Wang Y (2015) Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection. Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA, 8–10, June (2015); pp. 3982–3991

  44. Liu Y, Lew MS (2016) Learning relaxed deep supervision for better edge detection. Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, USA, June 26–July 1, pp.231–240

  45. Maninis KK, Pont-Tuset J, Arbeláez P (2016) Convolutional oriented boundaries. European conference on computer vision, Amsterdam, The Netherlands, 11–14, pp. 580–596

  46. Wang H, Jiao L, Wu H (2019) New inpainting algorithm based on simplified context encoders and multi-scale adversarial network. Procedia Comput Sci 147:254–263

    Article  Google Scholar 

  47. Liao L, Hu R, Xiao J (2018) Edge-aware context encoder for image inpainting. 2018 IEEE International Conference on Acoustics(ICASSP), Calgary, AB, USA, 15–20, pp. 3156–3160

  48. Mustafa A, Mikhailiuk A, Iliescu DA (2021) Training a Task-Specific Image Reconstruction Loss.Available online: http://128.84.4.34/abs/2103.14616. Access on 27 January 2021

  49. Zhang Y, Yin ZF, Li Y (2020) Celeba-spoof: Large-scale face anti-spoofing dataset with rich annotations. Proceedings of the IEEE conference on computer vision and pattern recognition, Glasgow, UK, 23–28, pp. 70–85

  50. Ting KM, Washio T, Wells JR (2017) Defying the gravity of learning curve: a characteristic of nearest neighbour anomaly detectors. Mach Learn 106(1):55–91

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Guangdong University of Technology, Guangzhou, China

    Gang Chen, Guipeng Zhang, Zhenguo Yang & Wenyin Liu

  2. Cyberspace Security Research Center, Peng Cheng Laboratory, Shenzhen, China

    Wenyin Liu

Authors
  1. Gang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guipeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenguo Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenyin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zhenguo Yang or Wenyin Liu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, G., Zhang, G., Yang, Z. et al. Multi-scale patch-GAN with edge detection for image inpainting. Appl Intell 53, 3917–3932 (2023). https://doi.org/10.1007/s10489-022-03577-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03577-2

Keywords

Search

Navigation