Abstract
This paper proposes new form of convolutional neural network that combines Convolutional Neural Networks (CNNs) and Conditional Random Fields (CRF) based probabilistic graphical modelling, which solve pixel level image labeling problem. In order to reduce the restrictions of deep learning techniques to delineate visual objects,the method fully integrates CRF modelling with CNNs, making it possible to train the whole deep network end-to-end with the usual back-propagation algorithm, avoiding offline post-processing methods for object delineation. Results show that the method is highly accurate and effective. The great result of the experiment have been achieved on the challenging Pascal VOC 2012 segmentation benchmark.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2015) Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR
Hoiem D, Efros AA, Hebert M (2008) Putting objects in perspective. Int J Comput Vis 80(1):3–15
He X, Zemel RS, Carreira Perpinan MA (2004) Multiscale conditional random fields for image labeling, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE, 2004, vol. 2, pp. II-695.
Krähenbühl P, Koltun V Efficient inferencein fully connected crfs with gaussian edge potentials, Advances in Neural Information Processing Systems (NIPS), pp. 109–117
Long J, Shelhamer E, Darrell T (2014) Fully convolutional networks for semantic segmentation. In IEEE transactions on Pattern Analysis & Machine. Intelligence 39(4):640–651
Mottaghi R, Chen X, Liu X, Cho NG, Lee SW,Fidler S, Urtasun R, Yuille A (2014) The role of contextfor object detection and semantic segmentation inthe wild. In IEEE CVPR
Nematollahiand M, Zhang XP (2015) A new robust context-based dense CRF model for image labeling, IEEE International Conference on Image Processing:5876–5880
Parikh D, Zitnick CL, Chen T (2008) From appearance to context-based recognition: Dense labeling in small images, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 1–8
PushmeetKohli P, Torr HS et al (2009) Robust higher order potentials for enforcing label consistency. Int J Comput Vis 82(3):302–324
Shotton J, Winn J, Rother C, Criminisi A (2009) Texton boost for image understanding: multi-class object recognition and segmentation by jointly modeling texture, layout, and context. Int J Comput Vis 81(1):2–23
Toyoda T, Hasegawa O (2008) Random field model for integration of local information and global information. IEEE Transactions on Pattern Analysisand Machine Intelligence 30(8):1483–1489
Vineet V, Warrell J, Sturgess P, Torr P (2012) Improved initialisation and gaussian mixturepairwise terms for dense random fields with meanfield inference,“in Proceedings of the British Machine Vision Conference. pp. 73.1–73.11, BMVA Press
Zhang Y, Chen T (2012) Efficient inferencefor fully-connected crfs with stationarity, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 582–589
Zheng S,Jayasumana S, Romera-Paredes B, Vineet V (2015) Conditional Random Fields as Recurrent Neural Networks. Proceedings of the IEEE International Conference on Computer Vision
Acknowledgements
The authors acknowledge the National Natural Science Foundation of China (Award Number:51405435).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Hu, K., Zhang, S. & Zhao, X. Context-based conditional random fields as recurrent neural networks for image labeling. Multimed Tools Appl 79, 17135–17145 (2020). https://doi.org/10.1007/s11042-019-7564-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-019-7564-x