Abstract
This paper investigates the reconstruction of Van Gogh’s drawings which have been degraded in the course of time due to aging problems, like ink fading and discoloration. Learning to predict the past and original appearances of degraded drawings can help to envisage how the artist’s work may have looked at the time of creation. In this paper, we use reproductions as reference information for the past appearances of drawings and consider the reconstruction of drawings as a pixel-wise prediction problem. We present an approach to automatically predict the past appearances of drawings. This approach brings together methods from multi-resolution image analysis and deep convolutional neural networks (CNNs) for addressing the task of pixel-wise prediction. Our experiments first investigate how scale affects prediction performance of the proposed multi-scale CNN framework and then demonstrate the reconstruction capability of the multi-scale CNN framework. The results demonstrate that the predictive reconstruction of degraded images is a feasible endeavor.
Similar content being viewed by others
References
Vellekoop, M., Jansen, L., Geldof, M., Hendriks, E., Tagle, A.: Van Goghs Studio Practice. Van Gogh Museum, Brussels (2013)
Berezhnoy, I.E., Postma, E.O., van den Herik, H.J.: Automatic extraction of brushstroke orientation from paintings. Mach. Vis. Appl. 20(1), 1–9 (2009)
Cornelis, B., Yang, H., Goodfriend, A., Ocon, N., Lu, J., Daubechies, I.: Removal of canvas patterns in digital acquisitions of paintings. IEEE Trans. Image Process. 26(1), 160–171 (2017)
Yan, C., Xie, H., Chen, J., Zha, Z., Hao, X., Zhang, Y., Dai, Q.: A fast uyghur text detector for complex background images. IEEE Trans. Multimed. 20(12), 3389–3398 (2018)
Zeng, Y., Tang, J., van der Lubbe, J., Loog, M.: Learning algorithms for digital reconstruction of Van Gogh’s drawings. In: 6th International Conference, EuroMed 2016, Cyprus, vol. 10058, pp. 322–333 (2016)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 1520–1528 (2015) (to appear)
Pappas, M., Pitas, I.: Digital color restoration of old paintings. IEEE Trans. Image Process. 9(2), 291–294 (2000)
Pei, S., Zeng, Y., Chang, C.: Virtual restoration of ancient Chinese paintings using color contrast enhancement and lacuna texture synthesis. IEEE Trans. Image Process. 13(3), 416–429 (2004)
Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: SIGGRAPH ’01: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, New York, NY, USA, pp. 327–340. ACM Press (2001)
Laffont, P., Ren, Z., Tao, X., Qian, C., Hays, J.: Transient attributes for high-level understanding and editing of outdoor scenes. ACM Trans. Graph. 33(4), 149 (2014)
Shih, Y., Paris, S., Durand, F., Freeman, W.T.: Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. 32(6), 200:1–200:11 (2013)
Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008)
Sun, J., Cao, W., Xu, Z., Ponce, J.: Learning a convolutional neural network for non-uniform motion blur removal. In: CVPR, pp. 769–777. IEEE Computer Society (2015)
Nah, S., Kim, T.H., Lee, K.M.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: CVPR, pp. 257–265. IEEE Computer Society (2017)
Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. In: CVPR, pp. 5261–5269. IEEE Computer Society (2015)
Lefkimmiatis, S.: Non-local color image denoising with convolutional neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 5882–5891(2017)
Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)
Wang, X., Yu, K., Dong, C., Loy, C.C.: Recovering realistic texture in image super-resolution by deep spatial feature transform. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 606–615 (2018)
Chaudhury, S., Roy, H.: Can fully convolutional networks perform well for general image restoration problems? In: MVA, pp. 254–257. IEEE (2017)
Kim, J., Lee, J.K., Lee, M.K: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 1646–1654 (2016)
Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3929–3938 (2017)
Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. CoRR (2015). arXiv:1511.05440
van Noord, N., Postma, E.O.: Learning scale-variant and scale-invariant features for deep image classification. Pattern Recognit. 61, 583–592 (2017)
Yoo, D., Park, S., Lee, J., So Kweon, I.: Multi-scale pyramid pooling for deep convolutional representation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2015)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Bach, F.R., Blei, D.M. (eds.) ICML, JMLR Workshop and Conference Proceedings, vol. 37, pp. 448–456. JMLR.org (2015)
Nair, V., Hinton, G.: Rectified linear units improve restricted Boltzmann machines. In: Frnkranz, J., Joachims, T. (eds.) Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814. Omni Press, Madison (2010)
Jian, B., Vemuri, B.C.: Robust point set registration using Gaussian mixture models. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1633–1645 (2011)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision (2016)
Xu, L., Ren, S.J., Liu, C.E., Jia, J.: Deep convolutional neural network for image deconvolution. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS’14, vol. 1, pp. 1790–1798. MIT Press, Cambridge (2014)
Nah, S., Kim, T., Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 257–265. IEEE Computer Society, Los Alamitos (2017)
Neevel, H., van Bommel, M., Vellekoop, M., Lingbeek, N.: From purple to brown—analysis of Van Gogh’s drawing inks; non-invasive identification of inks on Van Gogh’s drawings and letters. In: Proceedings of the 15th Triennial Conference, vol. 15, pp. 278–284 (2008)
Geerts, J.: Light damage on paper objects with coloured media; towards a differential sensitivity classification. MA thesis (2010)
Conover, D.M., Delaney, J.K., Loew, M.H.: Automatic registration and mosaicking of technical images of old master paintings. Appl. Phys. A 119(4), 1567–1575 (2015)
Acknowledgements
This research is a part of the REVIGO Project, supported by the Netherlands Organization for Scientific Research (NWO; Grant 323.54.004) in the context of the Science4Arts research program.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zeng, Y., van der Lubbe, J.C.A. & Loog, M. Multi-scale convolutional neural network for pixel-wise reconstruction of Van Gogh’s drawings. Machine Vision and Applications 30, 1229–1241 (2019). https://doi.org/10.1007/s00138-019-01047-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00138-019-01047-3