Skip to main content
SpringerLink
Log in

Restoration of damaged artworks based on a generative adversarial network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Ancient and contemporary artworks represent culture, heritage, and history. The artworks act as a bridge between the past and future of humankind. Preserving artwork is necessary for saving cultural heritage for future generations. However, artworks tend to deteriorate with time due to humidity, temperature, improper handling, and storage. Damages to artworks require unique restoration treatments, and, in many cases, they cannot be returned to their original state. Moreover, traditional artwork restoration methods are very costly and time-consuming. Further, the artwork restoration field is very complex as we not only require sharper and visually realistic restoration but also to preserve the artistic style and features of the artwork. The virtual restoration after digitizing the artworks can be very helpful in the actual restoration of the artworks. This paper works in this direction and proposes a novel generative adversarial network-based artwork restoration method that can digitally restore the damaged artwork, which can help in the physical restoration of the artworks. The proposed generative adversarial network uses a modified U-Net architecture for the generator part and pre-trained residual networks to build the generator's encoder part. Since the residual networks are pre-trained on millions of ImageNet images, they generate better feature embeddings out of the input artwork images, which in turn help in the generation of better-restored artwork images. The proposed artwork restoration method works in a single step and does not require the generation of masks. The proposed network has been evaluated on two different datasets using performance metrics such as peak signal-to-noise ratio, mean squared error, structural similarity index, and Fréchet inception distance. Results indicate that the proposed network outperforms the existing artwork restoration methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

The ‘Art images’ dataset used during the current study is available at the following URL: -

[https://www.kaggle.com/datasets/thedownhill/art-images-drawings-painting-sculpture-engraving] and ‘Best artwork of all times’ repository can be accessed from the following URL:-

[https://www.kaggle.com/datasets/ikarus777/best-artworks-of-all-time].

Notes

  1. https://www.kaggle.com/thedownhill/art-images-drawings-painting-sculpture-engraving

  2. https://www.kaggle.com/datasets/ikarus777/best-artworks-of-all-time?resource=download

References

  1. Adhikary A, Bhandari N, Markou E, Sachan S (2021) “ArtGAN: Artwork Restoration using Generative Adversarial Networks”, 2021 13th Int. Conf Adv Comput Intell ICACI 2021:199–206. https://doi.org/10.1109/ICACI52617.2021.9435888

    Article  Google Scholar 

  2. Bakkouri I, Afdel K (2020) Computer-aided diagnosis (CAD) system based on multi-layer feature fusion network for skin lesion recognition in dermoscopy images. Multimed Tools Appl vol. 79, 20483–20518. https://doi.org/10.1007/s11042-019-07988-1

  3. Bakkouri I, Afdel K (2022) “MLCA2F : Multi-Level Context Attentional Feature Fusion for COVID-19 lesion segmentation from CT scans”, Signal. Image Vid Process. https://doi.org/10.1007/s11760-022-02325-w

    Article  Google Scholar 

  4. Ballester C, Caselles V, Verdera J (2004) Disocclusion by joint interpolation of vector fields and gray levels. Multiscale Model Simul 2(1):80–123. https://doi.org/10.1137/S1540345903422458

    Article  MathSciNet  MATH  Google Scholar 

  5. Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans Graph 28(3):1–12. https://doi.org/10.1145/1531326.1531330

    Article  Google Scholar 

  6. Cao J, Zhang Z, Zhao A, Cui H, Zhang Q (2020) Ancient mural restoration based on a modified generative adversarial network. Herit Sci 8(1):1–14. https://doi.org/10.1186/s40494-020-0355-x

    Article  Google Scholar 

  7. Cornelis B et al (2013) Crack detection and inpainting for virtual restoration of paintings: The case of the Ghent Altarpiece. Signal Process 93(3):605–619. https://doi.org/10.1016/j.sigpro.2012.07.022

    Article  Google Scholar 

  8. Efros AA, Freeman WT (2005) “Image quilting for texture synthesis and transfer,” Proc. 28th Annu Conf Comput Graph Interact Tech, no. August, pp. 341–346, https://doi.org/10.1145/383259.383296

  9. Fei-Fei L, Deng J, Li K (2010) ImageNet: Constructing a large-scale image database. J Vis 9(8):1037–1037. https://doi.org/10.1167/9.8.1037

    Article  Google Scholar 

  10. Gatys L, Ecker A, Bethge M (2016) A Neural Algorithm of Artistic Style. J Vis 16(12):326. https://doi.org/10.1167/16.12.326

    Article  Google Scholar 

  11. Goodfellow I et al (2020) Generative adversarial networks. Commun ACM 63(11):139–144. https://doi.org/10.1145/3422622

    Article  MathSciNet  Google Scholar 

  12. Gupta V, Sambyal N, Sharma A, Kumar P (2021) Restoration of artwork using deep neural networks. Evol Syst 12(2):439–446. https://doi.org/10.1007/s12530-019-09303-7

    Article  Google Scholar 

  13. He K, Gkioxari G, Dollar P, Girshick R (2017) Mask R-CNN. Proc IEEE Int Conf Comput Vis 2017-Octob:2980–2988. https://doi.org/10.1109/ICCV.2017.322

    Article  Google Scholar 

  14. Iizuka S, Simo-Serra E, Ishikawa H (2017) “Globally and locally consistent image completion,” ACM Trans Graph, vol. 36, no. 4 https://doi.org/10.1145/3072959.3073659

  15. Jay F, Renou J-P, Voinnet O, Navarro L (2017) “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Jun-Yan,” Proc IEEE Int Conf Comput Vis, pp. 183–202, [Online]. https://doi.org/10.1007/978-1-60327-005-2_13

  16. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 9906:694–711. https://doi.org/10.1007/978-3-319-46475-6_43

    Article  Google Scholar 

  17. Knut Nicolaus E, Westphal C (1999) “The Restoration of Paintings.,” pp. 465–469, https://doi.org/10.1109/ICIEV.2018.8641016

  18. Levin A, Zomet A, Weiss Y (2003) Learning how to inpaint from global image statistics. Proc IEEE Int Conf Comput Vis 1:305–312. https://doi.org/10.1109/iccv.2003.1238360. Accessed 15 Nov 2021

  19. Li C, Wand M (2016) “Precomputed real-time texture synthesis with markovian generative adversarial networks,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9907 LNCS, pp. 702–716, https://doi.org/10.1007/978-3-319-46487-9_43

  20. Li J, Wang H, Deng Z, Pan M, Chen H (2021) Restoration of non-structural damaged murals in Shenzhen Bao’an based on a generator–discriminator network. Herit Sci 9(1):1–14. https://doi.org/10.1186/s40494-020-00478-w

    Article  Google Scholar 

  21. Liu G, Reda FA, Shih KJ, Wang T, Tao A, Catanzaro B (2018) Image Inpainting for Irregular Holes Using Partial Convolutions, vol. 11206. Springer International Publishing

  22. Liu G, Reda FA, Shih KJ, Wang TC, Tao A, Catanzaro B (2018) Image Inpainting for Irregular Holes Using Partial Convolutions, vol 11215. Springer International Publishing, LNCS

    Google Scholar 

  23. Marcelo Bertalmio CB, Sapiro G, Caselles V (2000) “Image inpainting,” Proc 27th Annu Conf Comput Graph Interact Tech pp. 417–424, https://doi.org/10.1055/s-0031-1298199

  24. Nazeri K, Ng E, Joseph T, Qureshi FZ, Ebrahimi M (2019) “EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning,”, [Online]. Available: http://arxiv.org/abs/1901.00212. Accessed 20 Jan 2022

  25. Park JY, Liu T, Wang MY, Zhu TC (2019) “GauGAN: semantic image synthesis with spatially adaptive normalization//ACM SIGGRAPH 2019. – July, 2019.– https://doi.org/10.1145/3306305.3332370,” p. 2019

  26. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) “Context Encoders: Feature Learning by Inpainting.” Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2016-Decem:2536–2544. https://doi.org/10.1109/CVPR.2016.278

    Article  Google Scholar 

  27. Sagong MC, Shin YG, Kim SW, Park S, Ko SJ (2019) “PEPSI : Fast image inpainting with parallel decoding network,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2019-June, pp. 11352–11360, https://doi.org/10.1109/CVPR.2019.01162

  28. Telea A (2004) An Image Inpainting Technique Based on the Fast Marching Method. J Graph Tools 9(1):23–34. https://doi.org/10.1080/10867651.2004.10487596

    Article  Google Scholar 

  29. Wan Z et al (2020) Old Photo Restoration via Deep Latent Space Translation. IEEE Trans Pattern Anal Mach Intell 45:2071–2087

    Article  Google Scholar 

  30. Wang HL et al. (2018) “Dunhuang mural restoration using deep learning,” SIGGRAPH Asia 2018 Tech. Briefs, SA. https://doi.org/10.1145/3283254.3283263. Accessed 10 Feb 2022

  31. Xie J, Xu L, Chen E (2012) Image denoising and inpainting with deep neural networks. Adv Neural Inf Process Syst 1(January 2012):341–349

    Google Scholar 

  32. Xu L, Ren JSJ, Liu C, Jia J (2014) Deep convolutional neural network for image deconvolution. Adv Neural Inf Process Syst 2(January):1790–1798

    Google Scholar 

  33. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) “Generative Image Inpainting with Contextual Attention,” Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit, pp. 5505–5514, https://doi.org/10.1109/CVPR.2018.00577

  34. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang T (2019) Free-form image inpainting with gated convolution. Proc IEEE Int Conf Comput Vis 2019-Octob:4470–4479. https://doi.org/10.1109/ICCV.2019.00457

    Article  Google Scholar 

  35. Zamir SW et al. (2021) “Multi-Stage Progressive Image Restoration,” 2021 IEEE/CVF Conf. Comput. Vis. Pattern Recognit, [Online]. Available: http://arxiv.org/abs/2102.02808

  36. Zeng Y, van der Lubbe JCA, Loog M (2019) Multi-scale convolutional neural network for pixel-wise reconstruction of Van Gogh’s drawings. Mach Vis Appl 30(7–8):1229–1241. https://doi.org/10.1007/s00138-019-01047-3

    Article  Google Scholar 

  37. Zou Z, Zhao P, Zhao X (2021) “Automatic segmentation , inpainting , and classification of defective patterns on ancient architecture using multiple deep learning algorithms,” no. March, pp. 1–18, https://doi.org/10.1002/stc.2742

  38. Zou Z, Zhao P, Zhao X (2021) Virtual restoration of the colored paintings on weathered beams in the Forbidden City using multiple deep learning algorithms. Adv Eng Inf 50(March):101421. https://doi.org/10.1016/j.aei.2021.101421

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Chandigarh College of Engineering and Technology, Sector 26, Chandigarh-160019, Chandigarh, India

    Praveen Kumar & Varun Gupta

Authors
  1. Praveen Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Varun Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Varun Gupta.

Ethics declarations

Competing Interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, P., Gupta, V. Restoration of damaged artworks based on a generative adversarial network. Multimed Tools Appl 82, 40967–40985 (2023). https://doi.org/10.1007/s11042-023-15222-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15222-2

Keywords

Search

Navigation