Skip to main content
SpringerLink
Log in

Artist-based painting classification using Markov random fields with convolution neural network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Determining the authorship of a painting image is a challenging task because paintings of an artist may not have a unique style and various artists may have similar painting styles. In this paper, we present a new approach to categorize digital painting images based on artist. We construct a multi-scale pyramid from a painting image to consider both globally and locally the information contained in one image. For each layer, we train a Convolutional Neural Network (CNN) model to determine the class label. To build the relationship within local image patches, we employ Markov random fields (MRFs) by optimizing the Gibbs energy function defined by (1) the data term measuring the compatibility of labeling with given data, and (2) the smoothness term penalizing assignments that label neighboring patches differently. A new fusion scheme is proposed to aggregate patch-level classification results. The proposed CNN-MRF method is validated using two challenging painting image datasets. Experimental results show that the proposed method is effective and achieves state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Chang Y-T, Cheng W-H, Wu B, Hua K-L (2017) Fashion world map: Understanding cities through streetwear fashion. In: Proceedings of the 25th ACM International Conference on Multimedia, pp 91–99

  2. Che Y, Song Y, Qi Y (2019) A novel framework of hand localization and hand pose estimation. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 2222–2226

  3. Cordero-Maldonado ML, Perathoner S, Van Der Kolk K-J, Boland R, Heins-Marroquin U, Spaink HP, Meijer AH, Crawford AD, De Sonneville J (2019) Deep learning image recognition enables efficient genome editing in zebrafish by automated injections. PloS One 14:e0202377

    Article  Google Scholar 

  4. Feng F, Wang X, Li R (2014) Cross-modal retrieval with correspondence autoencoder. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp 7–16

  5. Freeman WT, Pasztor EC, Carmichael OT (2000) Learning low-level vision. Int JComput Vision 40:25–47

    Article  Google Scholar 

  6. Guo J, Song B, Zhang P, Ma M, Luo W (2019) Affective video content analysis based on multimodal data fusion in heterogeneous networks. Inform Fusion 51:224–232

    Article  Google Scholar 

  7. Guo Y, Liu Y, Bakker EM, Guo Y, Lew MS (2018) CNN-RNN: A large-scale hierarchical image classification framework. Multimed Tools Appl 77:10251–10271

    Article  Google Scholar 

  8. Hammersley JM, Clifford P (1971) Markov fields on finite graphs and lattices. Unpublished manuscript, pp 46

  9. Hua K-L, Hsu C-H, Hidayati SC, Cheng W-H, Chen Y-J (2015) Computer-aided classification of lung nodules on computed tomography images via deep learning technique. OncoTargets and therapy, pp 8

  10. Jangtjik KA, Ho T-T, Yeh M-C, Hua K-L (2017) A CNN-LSTM framework for authorship classification of paintings. In: 2017 IEEE International Conference on Image Processing (ICIP), IEEE, pp 2866–2870

  11. Jangtjik KA, Yeh M-C, Hua K-L (2016) Artist-based classification via deep learning with multi-scale weighted pooling. In: In: Proceedings of the 24th ACM International Conference on Multimedia, pp 635–639

  12. Kalliatakis G, Ehsan S, Leonardis A, Fasli M, McDonald-Maier KD (2019) Exploring object-centric and scene-centric CNN features and their complementarity for human rights violations recognition in images. IEEE Access 7:10045–10056

    Article  Google Scholar 

  13. Katib I, Medhi D (2011) A study on layer correlation effects through a multilayer network optimization problem. In: Proceedings of the 23rd International Teletraffic Congress, International Teletraffic Congress, pp 31–38

  14. Ke J, Peng Y, Liu S, Sun Z, Wang X (2019) A novel grouped sparse representation for face recognition. Multimed Tools Appl 78:7667–7689

    Article  Google Scholar 

  15. Kelek MO, Calik N, Yildirim T (2019) Painter classification over the novel art painting data set via the latest deep neural networks. Procedia Comput Sci 154:369–376

    Article  Google Scholar 

  16. Khan S, Islam N, Jan Z, Din IUd, Rodrigues JJPC (2019) A novel deep learning based framework for the detection and classification of breast cancer using transfer learning. Pattern Recogn Lett 125:1–6

    Article  Google Scholar 

  17. Kim D, Yoon K-j (2012) High-quality depth map up-sampling robust to edge noise of range sensors. In: 2012 19th IEEE International Conference on Image Processing, IEEE, pp 553-556

  18. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp 1097–1105

  19. Kumar S, Tyagi A, Sahu T, Shukla P, Mittal A (2018) Indian art form recognition using convolutional neural networks. In: 2018 5th International Conference on Signal Processing and Integrated Networks (SPIN), IEEE, pp 800–804

  20. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1:541–551

    Article  Google Scholar 

  21. Lee JY (2019) Deep learning ensemble with data augmentation using a transcoder in visual description. Multimedia Tools and Applications, pp 1–13

  22. Li P, Zhao L, Duanqing X, Lu D (2019) Optimal transport of deep feature for image style transfer. In: Proceedings of the 2019 4th International Conference on Multimedia Systems and Signal Processing, ACM, pp 167–171

  23. Liu Z, Li X, Luo P, Loy CC, Tang X (2017) Deep learning markov random field for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 40:1814–1828

    Article  Google Scholar 

  24. Lo K-H, Hua K-L, Wang Y-CF (2013) Depth map super-resolution via Markov random fields without texture-copying artifacts. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, pp 1414–1418

  25. Lu J, Min D, Pahwa RS, Do MN (2011) A revisit to MRF-based depth map super-resolution and enhancement. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 985-988

  26. PaintingDb (2015) PaintingDb Fastest growing art gallery in the web, http://www.paintingdb.com

  27. Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans Knowl Data Eng 22:1345–1359

    Article  Google Scholar 

  28. Peng K-C, Chen T (2015) Cross-layer features in convolutional neural networks for generic classification tasks. In: 2015 IEEE International Conference on Image Processing (ICIP), IEEE, pp 3057–3061

  29. Perez P (1998) Markov random fields and images pp. 31 IRISA

  30. Qi H, Hughes S (2011) A new method for visual stylometry on impressionist paintings. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pp 2036–2039

  31. Qiu Z, Yan F, Zhuang Y, Leung H (2019) Outdoor Semantic Segmentation for UGVs Based on CNN and Fully Connected CRFs. IEEE Sensors J 19:4290–4298

    Article  Google Scholar 

  32. Sanchez-Riera J, Srinivasan K, Hua K-L, Cheng W-H, Anwar Hossain M, Alhamid MF (2017) Robust RGB-d hand tracking using deep learning priors. IEEE Trans Circuits Syst Video Technol 28:2289–2301

    Article  Google Scholar 

  33. Sandoval C, Pirogova E, Lech M (2019) Two-stage deep learning approach to the classification of fine-art paintings. IEEE Access 7:41770–41781

    Article  Google Scholar 

  34. Sudharshan PJ, Petitjean C, Spanhol F, Oliveira LE, Heutte L, Honeine P (2019) Multiple instance learning for histopathological breast cancer image classification. Expert Syst Appl 117:103–111

    Article  Google Scholar 

  35. Sun M, Zhang D, Ren J, Wang Z, Jin JS (2015) Brushstroke based sparse hybrid convolutional neural networks for author classification of Chinese ink-wash paintings. In: 2015 IEEE International Conference on Image Processing (ICIP), IEEE, pp 626–630

  36. Tan WR, Chan CS, Aguirre HE, Tanaka K (2016) Ceci n’est pas une pipe: A deep convolutional network for fine-art paintings classification. In: 2016 IEEE international conference on image processing (ICIP), IEEE, pp 3703–3707

  37. Wang W, Chen G, Chen H, Dinh TTA, Gao J, Ooi BC, Tan K-L, Wang S, Zhang M (2014) Deep learning at scale and at ease. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 12:69

    Google Scholar 

  38. Wang W, Ooi BC, Yang X, Zhang D, Zhuang Y (2014) Effective multi-modal retrieval based on stacked auto-encoders. Proceedings of the VLDB Endowment 7:649–660

    Article  Google Scholar 

  39. WikiArt (2016) WikiArt the online home for visual arts from all around the world

  40. Yang X, Ye Y, Li X, Lau RYK, Zhang X, Huang X (2018) Hyperspectral image classification with deep learning models. IEEE Trans Geosci Remote Sens 56:5408–5423

    Article  Google Scholar 

  41. Zhang L, Wang S, Liu B (2018) Deep learning for sentiment analysis A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8:e1253

    Google Scholar 

  42. Zhao S, Yao H, Jiang X, Sun X (2015) Predicting discrete probability distribution of image emotions. In: 2015 IEEE International Conference on Image Processing (ICIP), IEEE, pp 2459–2463

  43. Zhong S-h, Huang X, Xiao Z (2019) Fine-art painting classification via two-channel dual path networks. In: International Journal of Machine Learning and Cybernetics, Springer, pp 1–16

  44. Zhong S-H, Liu Y, Hua KA (2016) Field effect deep networks for image recognition with incomplete data. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 12:52

    Google Scholar 

  45. Zhong S-H, Liu Y, Liu Y (2011) Bilinear deep learning for image classification. In: Proceedings of the 19th ACM international conference on Multimedia, pp 343–352

  46. Zoph B, Vasudevan V, Shlens J, Le QV (2018) Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 8697–8710

Download references

Acknowledgements

This work was supported in part by the NTUST-MMH Joint Research Program (NTUST-MMH-No 10601) and the Ministry of Science and Technology (108-2221-E-011-116, 108-2218-E-011-026).

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan

    Kai-Lung Hua, Trang-Thi Ho & Kevin-Alfianto Jangtjik

  2. MacKay Memorial Hospital, Taipei, Taiwan

    Yu-Jen Chen

  3. Department of Computer Science and Information Engineering, National Taiwan Normal University, Taipei, Taiwan

    Mei-Chen Yeh

Authors
  1. Kai-Lung Hua
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Trang-Thi Ho
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kevin-Alfianto Jangtjik
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu-Jen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mei-Chen Yeh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mei-Chen Yeh.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hua, KL., Ho, TT., Jangtjik, KA. et al. Artist-based painting classification using Markov random fields with convolution neural network. Multimed Tools Appl 79, 12635–12658 (2020). https://doi.org/10.1007/s11042-019-08547-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08547-4

Keywords

Search

Navigation