Abstract
Mural deterioration easily destroys valuable paintings and must be monitored frequently for preventive protection. Deterioration detection in mural images is often manually labeled and is a preprocessing step for mural protection and restoration. Many deterioration forms are commonly invisible with only one lighting condition because mural deterioration is caused by changes in the material and plaster layer. This study addresses mural deterioration detection through a multi-path convolutional neural network (CNN), which takes images of a scene with multiple lightings as inputs and generates a binary map that indicates deterioration regions. We design an eight-path CNN in which seven paths are utilized for basic feature extraction from lighted images, and the remaining path is responsible for cross feature fusion. This mechanism enables our method to not only identify suitable features for different lightings but also utilize these features collaboratively through cross feature fusion. Furthermore, we build two realistic mural deterioration datasets of real-world mural deterioration and briquettes that simulate the cave deterioration. Extensive experiments verify the effectiveness and efficiency of our method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We denote the filter’s size by \(H \times W \times \#Input \times \#Output\), where H denotes filter’s height, W denotes filter’s width, \(\#Input\) denotes input dimension and \(\#Output\) denotes output dimension.
References
Bai M, Luo W, Kundu K, Urtasun R (2016) Exploiting semantic information and deep matching for optical flow. In: European conference on computer vision. https://doi.org/10.1007/978-3-319-46466-4_10
Breiman L (2001) Random forests. Mach Learn 45(1):5–32. https://doi.org/10.1023/A:1010933404324
Chang C-C, Lin C-J (2014) Libsvm : a library for support vector machines. ACM Trans Intell Syst and Technol 2(3):1–27. https://doi.org/10.1145/1961189.1961199
Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2015) Semantic image segmentation with deep convolutional nets and fully connected CRFs. In: International conference on learning representations. arXiv:1412.7062
Deng J, Dong W, Socher R, Li LJ, Li K, Li FF (2009) Imagenet: A large-scale hierarchical image database. In: Computer vision and pattern recognition, pp 248–255. https://doi.org/10.1109/CVPRW.2009.5206848
Dosovitskiy A, Fischery P, Ilg E, Hausser P, Hazirbas C, Golkov V, Smagt PVD, Cremers D, Brox T (2015) Flownet: Learning optical flow with convolutional networks. In: International conference on computer vision. https://doi.org/10.1109/ICCV.2015.316
Farabet C, Couprie C, Najman L, LeCun Y (2013) Learning hierarchical features for scene labeling. IEEE Trans Pattern Anal Mach Intell 35(8):1915–1929. https://doi.org/10.1109/TPAMI.2012.231
Feichtinger HG, Strohmer T (1998) Gabor analysis and algorithms: theory and applications. Birkhauser, Boston
Felzenszwalb PF, Girshick RB, McAllester D, Ramanan D (2010) Object detection with discriminatively trained part-based models. IEEE Trans Pattern Anal Mach Intell 32(9):1627–1645. https://doi.org/10.1109/TPAMI.2009.167
Feng W, Liu ZQ (2008) Region-level image authentication using bayesian structural content abstraction. IEEE Trans Image Process 17(12):2413–2424. https://doi.org/10.1109/TIP.2008.2006435
Feng W, Jia J, Liu ZQ (2010) Self-validated labeling of markov random fields for image segmentation. IEEE Trans Pattern Anal Mach Intell 32(10):1871–1887. https://doi.org/10.1109/TPAMI.2010.24
Feng W, Tian FP, Zhang Q, Zhang N, Wan L, Sun J (2015) Fine-grained change detection of misaligned scenes with varied illuminations. In: International conference on computer vision. https://doi.org/10.1109/ICCV.2015.149
Feng W, Yang Y, Liang W, Yu C (2016) Tone-mapped mean-shift based environment map sampling. IEEE Trans Vis Comput Graph 22(9):1077–2626. https://doi.org/10.1109/TVCG.2015.2500236
Gould S, Fulton R, Koller D (2009) Decomposing a scene into geometric and semantically consistent regions. In: International conference on computer vision. https://doi.org/10.1109/ICCV.2009.5459211
Guo Q, Feng W, Zhou C, Huang R, Wan L, Wang S (2017a) Learning dynamic siamese network for visual object tracking. In: International conference on computer vision
Guo Q, Feng W, Zhou C, Pun CM, Wu B (2017) Structure-regularized compressive tracking with online data-driven sampling. IEEE Trans Image Process 26(12):5692–5705. https://doi.org/10.1109/TIP.2017.2745205
Hariharan B, Arbeláez P, Girshick R, Malik J (2014) Simultaneous detection and segmentation. In: European conference on computer vision. https://doi.org/10.1007/978-3-319-10584-0_20
Hariharan B, Arbeláez P, Girshick R, Malik J (2015) Hypercolumns for object segmentation and fine-grained localization. In: Computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2015.7298642
He X, Zemel RS, Carreira-Perpiñán MÁ (2004) Multiscale conditional random fields for image labeling. In: Computer vision and pattern recognition
Huang R, Feng W, Sun J (2017) Color feature reinforcement for co-saliency detection without single saliency residuals. IEEE Signal Process Lett 24(5):569–573. https://doi.org/10.1109/LSP.2017.2681687
Huang R, Feng W, Wang Z, Fan M, Wan L, Sun J (2017b) Learning to detect fine-grained change under variant imaging conditions. In: International conference on computer vision workshop on E-Heritage
Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. In: ACM MM. https://doi.org/10.1145/2647868.2654889
Koltun V (2011) Efficient inference in fully connected crfs with gaussian edge potentials. In: International conference on neural information processing systems, pp 109–117
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: International conference on neural information processing systems. https://doi.org/10.1145/3065386
Liu C, Yuen J, Torralba A (2009) Nonparametric scene parsing: label transfer via dense scene alignment. In: Computer vision and pattern recognition. https://doi.org/10.1145/3065386
Liu J, Lu D (2007) Knowledge based lacunas detection and segmentation for ancient paintings. In: International conference on virtual systems and multimedia, pp 121–131
Liu S, Liang X, Liu L, Shen X, Yang J, Xu C, Lin L, Cao X, Yan S (2015) Matching-cnn meets knn: Quasi-parametric human parsing. In: Computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2015.7298748
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2015.7298965
Mostajabi M, Yadollahpour P, Shakhnarovich G (2015) Feedforward semantic segmentation with zoom-out features. In: Computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2015.7298959
Papandreou G, Chen LC, Murphy K, Yuille AL (2015) Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In: International conference on computer vision. https://doi.org/10.1109/ICCV.2015.203
Pinchin S (2013) Historical perspectives on preventive conservation. J Archit Conserv 4119(1):1–3
Shotton J, Winn J, Rother C, Criminisi A (2006) Textonboost: joint appearance, shape and context modeling for multi-class object recognition and segmentation. In: European conference on computer vision. https://doi.org/10.1007/11744023_1
Tighe J, Lazebnik S (2010) Superparsing: scalable nonparametric image parsing with superpixels. In: European conference on computer vision. https://doi.org/10.1007/978-3-642-15555-0_26
Tighe J, Lazebnik S (2013) Finding things: image parsing with regions and per-exemplar detectors. In: Computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2013.386
Wirilander H (2012) Preventive conservation: a key method to ensure cultural heritages authenticity and integrity in preservation process. E-Conserv Mag 6(24):164–176
Zhang H, Wang J, Tan P, Wang J, Quan L (2013) Learning crfs for image parsing with adaptive subgradient descent. In: International conference on computer vision. https://doi.org/10.1109/ICCV.2013.382
Zheng S, Jayasumana S, Romera-Paredes B, Vineet V, Su Z, Du D, Huang C, Torr PH (2015) Conditional random fields as recurrent neural networks. In: International conference on computer vision. https://doi.org/10.1109/ICCV.2015.179
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 61671325 and 61572354, and in part by the National Science and Technology Support Project 2013BAK01B01.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Huang, R., Feng, W., Fan, M. et al. Learning multi-path CNN for mural deterioration detection. J Ambient Intell Human Comput 11, 3101–3108 (2020). https://doi.org/10.1007/s12652-017-0656-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-017-0656-4