skip to main content
research-article

Inpainting Digital Dunhuang Murals with Structure-Guided Deep Network

Published: 06 December 2022 Publication History

Abstract

Inpainting deteriorated regions in digital Dunhuang murals is important for Dunhuang mural content preservation. Algorithms of mural image inpainting help simplify the digital restoration process of the deteriorated murals. Most of the existing algorithms can restore plausible content for homogeneous missing regions in Dunhuang mural images. However, they often fail to fill accurate color in missing regions that contain complex structures, which is mainly due to the neglect of color relevance between positions in the missing structural region and the non-missing color regions. In this article, we propose a deep learning–based, structure-guided inpainting method for the Dunhuang mural image, which utilizes relevant color information in deep features to improve the color inpainting quality for structural regions. Specifically, we design a structure-guided feature refinement module, which explicitly leverages color relevance implied in structure information to select relevant features for refining features in the missing region. In addition, we propose a multi-step scheme for feature refinement to better propagate non-missing region feature information to the missing region. We conduct experiments on Dunhuang660 and Dunhuang No. 7 Grotto datasets. The results demonstrate that our proposed method can achieve improved color inpainting quality for missing structural regions in Dunhuang mural images.

References

[1]
Dunhuang Academy. 2021. Digital Dunhuang. Retrieved November 15, 2021 from https://www.e-dunhuang.com/.
[2]
Marcelo Bertalmio, Andrea L. Bertozzi, and Guillermo Sapiro. 2001. Navier-Stokes, fluid dynamics, and image and video inpainting. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’01), Vol. 1, Kauai, HI, USA. IEEE, I–I.
[3]
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. 2000. Image inpainting. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, Louisiana, USA. ACM, 417–424.
[4]
Folkmar Bornemann and Tom März. 2007. Fast image inpainting based on coherence transport. Journal of Mathematical Imaging and Vision 28, 3 (2007), 259–278.
[5]
Jianfang Cao, Yanfei Li, Qi Zhang, and Hongyan Cui. 2019. Restoration of an ancient temple mural by a local search algorithm of an adaptive sample block. Heritage Science 7, 1 (2019), 1–14.
[6]
Jianfang Cao, Zibang Zhang, Aidi Zhao, Hongyan Cui, and Qi Zhang. 2020. Ancient mural restoration based on a modified generative adversarial network. Heritage Science 8, 1 (2020), 1–14.
[7]
Tony F. Chan and Jianhong Shen. 2001. Nontexture inpainting by curvature-driven diffusions. Journal of Visual Communication and Image Representation 12, 4 (2001), 436–449.
[8]
Long Chen, Jianda Chen, Qin Zou, Kai Huang, and Qingquan Li. 2017. Multi-view feature combination for ancient paintings chronological classification. Journal on Computing and Cultural Heritage 10, 2, Article 7 (March2017), 15 pages.
[9]
Ming Chen, Xudong Zhao, and Duanqing Xu. 2019. Image inpainting for digital Dunhuang murals using partial convolutions and sliding window method. In Journal of Physics: Conference Series, Vol. 1302. IOP Publishing, 1–6.
[10]
Irina-Mihaela Ciortan, Sony George, and Jon Yngve Hardeberg. 2021. Colour-balanced edge-guided digital inpainting: Applications on artworks. Sensors 21, 6 (2021), 1–21.
[11]
Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13, 9 (2004), 1200–1212.
[12]
X. Y. Fu, Y. Han, Z. J. Sun, X. J. Ma, and Y. Q. Xu. 2017. Line-drawing enhanced interactive mural restoration for Dunhuang Mogao Grottoes. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 4 (2017), 99–106.
[13]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Communications of the ACM 63, 11 (2020), 139–144.
[14]
Wu Hung. 1992. Reborn in paradise: A case study of Dunhuang sutra painting and its religious, ritual and artistic context. Orientations 23 (1992), 52–60.
[15]
Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and locally consistent image completion. ACM Transactions on Graphics 36, 4, Article 107 (July2017), 14 pages.
[16]
Salinee Jaidilert and Ghulam Farooque. 2018. Crack detection and images inpainting method for Thai mural painting images. In IEEE 3rd International Conference on Image, Vision and Computing (ICIVC’18), Chongqing, China. IEEE, 143–148.
[17]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, Amsterdam, The Netherlands. Springer, 694–711.
[18]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. ICLR. 2015. San Diego, CA, USA.
[19]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25 (2012), 1097–1105.
[20]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436–444.
[21]
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11 (1998), 2278–2324.
[22]
Jiao Li, Huan Wang, Zhiqin Deng, Mingtao Pan, and Honghai Chen. 2021. Restoration of non-structural damaged murals in Shenzhen Bao’an based on a generator–discriminator network. Heritage Science 9, 1 (2021), 1–14.
[23]
Jiamin Li, Hui Zhang, Zaixuan Fan, Xiang He, Shimin He, Mingyuan Sun, Yimin Ma, Shi Qiang Fang, Huabing Zhang, and Bingjian Zhang. 2013. Investigation of the renewed diseases on murals at Mogao Grottoes. Heritage Science 1 (102013), 31.
[24]
Xinghua Li, Huanfeng Shen, Liangpei Zhang, Hongyan Zhang, Qiangqiang Yuan, and Gang Yang. 2014. Recovering quantitative remote sensing products contaminated by thick clouds and shadows using multitemporal dictionary learning. IEEE Transactions on Geoscience and Remote Sensing 52, 11 (2014), 7086–7098.
[25]
Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV’18), Munich, German. Springer, 85–100.
[26]
Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, and Jing Liao. 2021. PD-GAN: Probabilistic diverse GAN for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21), Virtual. IEEE, 9371–9381.
[27]
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. 2019. EdgeConnect: Structure guided image inpainting using edge prediction. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW’19), Seoul, Korea. IEEE, 3265–3274.
[28]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32 (2019), 8026–8037.
[29]
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Nevada, USA. IEEE, 2536–2544.
[30]
Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. 2021. Generating diverse structure for image inpainting with hierarchical VQ-VAE. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21), Virtual. IEEE, 10775–10784.
[31]
Pulak Purkait and Bhabatosh Chanda. 2012. Digital restoration of damaged mural images. In Proceedings of the 8th Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP’12), Mumbai, India. ACM, New York, NY, Article 49, 8 pages.
[32]
Zhenping Qiang, Libo He, Yaqiong Chen, and Dan Xu. 2017. Research on mural inpainting method based on MCA image decomposition. In Proceedings of the 2nd International Conference on Intelligent Information Processing (IIP’17), Bangkok Thailand. ACM, New York, NY, Article 23, 9 pages.
[33]
Stefan Roth and Michael J. Black. 2005. Fields of experts: A framework for learning image priors. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 2. IEEE, 860–867.
[34]
Shankar Setty and Uma Mudenagudi. 2018. Region of interest-based 3D inpainting of cultural heritage artifacts. Journal on Computing and Cultural Heritage 11, 2, Article 9 (May2018), 21 pages.
[35]
Jianhong Shen and Tony F. Chan. 2002. Mathematical models for local nontexture inpaintings. SIAM Journal on Applied Mathematics 62, 3 (2002), 1019–1043.
[36]
I. E. Sobel. 1970. Camera Models and Machine Perception. Stanford University Press. https://books.google.com.hk/books?id=RrGpnQEACAAJ.
[37]
Maitreya Suin, Kuldeep Purohit, and A. N. Rajagopalan. 2021. Distillation-guided image inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’21), Virtual. IEEE, 2481–2490.
[38]
Meijun Sun, Dong Zhang, Zheng Wang, Jinchang Ren, Bolong Chai, and Jizhou Sun. 2015. What’s wrong with the murals at the Mogao Grottoes: a near-infrared hyperspectral imaging method. Scientific Reports 5, 1 (2015), 1–10.
[39]
Carlo Tomasi and Roberto Manduchi. 1998. Bilateral filtering for gray and color images. In 6th International Conference on Computer Vision (IEEE Cat. No. 98CH36271), Bombay, Indias. IEEE, 839–846.
[40]
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Utah, USA. IEEE, 9446–9454.
[41]
Huan Wang, Qingquan Li, and Sen Jia. 2020. A global and local feature weighted method for ancient murals inpainting. International Journal of Machine Learning and Cybernetics 11, 6 (2020), 1197–1216.
[42]
Huan Wang, Qingquan Li, and Qin Zou. 2019. Inpainting of Dunhuang murals by sparsely modeling the texture similarity and structure continuity. Journal on Computing and Cultural Heritage 12, 3 (2019), 1–21.
[43]
Nianyi Wang, Weilan Wang, Wenjin Hu, Aaron Fenster, and Shuo Li. 2021. Thanka mural inpainting based on multi-scale adaptive partial convolution and stroke-like mask. IEEE Transactions on Image Processing 30 (2021), 3720–3733.
[44]
X. Wang, N. Song, Z. Lu, and Y. Jiang. 2017. Understanding subjects contained in Dunhuang mural images for deep semantic annotation. Journal of Documentation 74, 2 (2017), 333–353.
[45]
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612.
[46]
Thomas Weber, Heinrich Hußmann, Zhiwei Han, Stefan Matthes, and Yuanting Liu. 2020. Draw with me: Human-in-the-loop for image restoration. In Proceedings of the 25th International Conference on Intelligent User Interfaces, Cagliari Italy. ACM, 243–253.
[47]
Huang Wei and Wang Shuwen. 2010. Dunhuang murals inpainting based on image decomposition. In 3rd International Conference on Computer Science and Information Technology, Vol. 2, Chengdu, China. IEEE, 397–400.
[48]
Tianxiu Yu, Cong Lin, Shijie Zhang, Shaodi You, Xiaohong Ding, Jian Wu, and Jiawan Zhang. 2019. End-to-end partial convolutions neural networks for Dunhuang grottoes wall-painting restoration. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW’19), Seoul, Korea. IEEE, 1447–1455.
[49]
Tianxiu Yu, Shijie Zhang, Cong Lin, Shaodi You, Jian Wu, Jiawan Zhang, Xiaohong Ding, and Huili An. 2019. Dunhuang grottoes painting dataset and benchmark. CoRR abs/1907.04589 (2019). arXiv:1907.04589http://arxiv.org/abs/1907.04589.
[50]
Yingchen Yu, Fangneng Zhan, Shijian Lu, Jianxiong Pan, Feiying Ma, Xuansong Xie, and Chunyan Miao. 2021. WaveFill: A wavelet-based generation network for image inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV’21), Virtual. IEEE, 14114–14123.
[51]
Yuan Zeng and Yi Gong. 2018. Nearest neighbor based digital restoration of damaged ancient Chinese paintings. In IEEE 23rd International Conference on Digital Signal Processing (DSP’18), Shanghai, China. IEEE, 1–5.
[52]
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 6 (2017), 1452–1464.
[53]
Qin Zou, Yu Cao, Qingquan Li, Chuanhe Huang, and Song Wang. 2014. Chronological classification of ancient paintings using appearance and shape features. Pattern Recognition Letters 49 (2014), 146–154.

Cited By

View all
  • (2025)ArtDiff: Integrating IoT and AI to enhance precision in ancient mural restorationAlexandria Engineering Journal10.1016/j.aej.2024.09.120111(511-520)Online publication date: Jan-2025
  • (2024)The generation method of orthophoto expansion map of arched dome mural based on three-dimensional fine color modelHeritage Science10.1186/s40494-024-01529-212:1Online publication date: 21-Nov-2024
  • (2024)Virtual restoration of ancient tomb murals based on hyperspectral imagingHeritage Science10.1186/s40494-024-01501-012:1Online publication date: 22-Nov-2024
  • Show More Cited By

Index Terms

  1. Inpainting Digital Dunhuang Murals with Structure-Guided Deep Network

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Journal on Computing and Cultural Heritage
    Journal on Computing and Cultural Heritage   Volume 15, Issue 4
    December 2022
    483 pages
    ISSN:1556-4673
    EISSN:1556-4711
    DOI:10.1145/3572828
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 06 December 2022
    Online AM: 25 July 2022
    Accepted: 19 April 2022
    Revised: 05 March 2022
    Received: 19 November 2021
    Published in JOCCH Volume 15, Issue 4

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Dunhuang mural
    2. digital mural inpainting
    3. deep learning
    4. structure-guided

    Qualifiers

    • Research-article
    • Refereed

    Funding Sources

    • National Key R&D Program of China
    • National Natural Science Foundation of China
    • Guangdong Provincial Key Laboratory of Human Digital Twin
    • Guangzhou city science and technology research projects

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)245
    • Downloads (Last 6 weeks)29
    Reflects downloads up to 21 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)ArtDiff: Integrating IoT and AI to enhance precision in ancient mural restorationAlexandria Engineering Journal10.1016/j.aej.2024.09.120111(511-520)Online publication date: Jan-2025
    • (2024)The generation method of orthophoto expansion map of arched dome mural based on three-dimensional fine color modelHeritage Science10.1186/s40494-024-01529-212:1Online publication date: 21-Nov-2024
    • (2024)Virtual restoration of ancient tomb murals based on hyperspectral imagingHeritage Science10.1186/s40494-024-01501-012:1Online publication date: 22-Nov-2024
    • (2024)Extraction and restoration of scratched murals based on hyperspectral imaging—a case study of murals in the East Wall of the sixth grotto of Yungang Grottoes, Datong, ChinaHeritage Science10.1186/s40494-024-01215-312:1Online publication date: 15-Apr-2024
    • (2024)Image Restoration Technology of Tang Dynasty Tomb Murals Using Adversarial Edge LearningJournal on Computing and Cultural Heritage 10.1145/367498417:3(1-11)Online publication date: 15-Jul-2024
    • (2024)Art Image Inpainting With Style-Guided Dual-Branch Inpainting NetworkIEEE Transactions on Multimedia10.1109/TMM.2024.337496326(8026-8037)Online publication date: 12-Mar-2024
    • (2024)Edge-Guided Mural Image Inpainting by Integrating Local and Global Information and Multiple Color Spaces2024 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME57554.2024.10687486(1-6)Online publication date: 15-Jul-2024
    • (2024)AGD-GAN: Adaptive Gradient-Guided and Depth-supervised generative adversarial networks for ancient mural sketch extractionExpert Systems with Applications10.1016/j.eswa.2024.124639255(124639)Online publication date: Dec-2024
    • (2023)Progressive Dunhuang murals inpainting based on recurrent feature reasoning networkJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-23032045:4(6667-6678)Online publication date: 1-Jan-2023
    • (2023)A virtual restoration network of ancient murals via global–local feature extraction and structural information guidanceHeritage Science10.1186/s40494-023-01109-w11:1Online publication date: 15-Dec-2023
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    Full Text

    HTML Format

    View this article in HTML Format.

    HTML Format

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media