Skip to main content
SpringerLink
Log in

Labeling of partially occluded regions via the multi-layer CRF

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This work proposes a general multi-layer framework for image labeling, which targets the challenging problem of classifying the occluded parts of the 3D scene depicted in a 2D image. Our framework is based on the mixed graphical models, which explicitly encode causal relationship between the visible and occluded regions. Unlike other image labeling techniques where a single label is determined for each pixel, layered model assigns multiple labels to pixels. We propose a novel “Multi-Layer-CRF” framework that allows for the integration of sophisticated occlusion potentials into the model and enables the automatic inference of the layer decomposition. We use a special message-passing algorithm to perform maximum a posterior inference on mixed graphs and demonstrate the ability to infer the correct labels of occluded regions in both the aerial near-vertical dataset and urban street-view dataset. It is shown to increase the classification accuracy in occluded areas significantly.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. We designate \(\mathbb {L}^{z}_{+} \equiv \mathbb {L}^{z}\cup \{void\}\).

  2. These values were taken from the presentation, available online: http://aqua.cs.uiuc.edu/site/files/eccv2012_ruiqi.pptx

References

  1. Bileschi SM (2006) Streetscenes: towards scene understanding in still images. Ph.D. thesis MIT

  2. Boykov Y, Jolly MP (2001) Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. In: Proceedings of ICCV, pp 105–112

  3. Breiman L. (2001) Random forests. Mach Learn

  4. Chen YT, Liu X, Yang MH (2015) Multi-instance object segmentation with occlusion handling. In: Proceedings of CVPR, pp 3470–3478

  5. Cramer M (2010) The DGPF test on digital aerial camera evaluation - overview and test design. PFG 2(2010):73–82

    Google Scholar 

  6. Criminisi A, Shotton J (2013) Decision forests in computer vision and medical image analysis. Springer, Berlin

    Book  Google Scholar 

  7. Duda R, Hart P (1972) Use of the hough transformation to detect lines and curves in pictures. Commun ACM 15(1):11–15

    Article  Google Scholar 

  8. Felzenszwalb PF, Girshick RB, McAllester D, Ramanan D (2010) Object detection with discriminatively trained part-based models. IEEE Trans Pattern Anal Mach Intell 32(9):1627–1645

    Article  Google Scholar 

  9. Frydenberg M (1990) The chain graph Markov property. Scand J Stat 17:333–353

    MathSciNet  MATH  Google Scholar 

  10. Guo R, Hoiem D (2012) Beyond the line of sight: labeling the underlying surfaces. In: Proceedings of ECCV, pp 761–774

  11. Guo R, Hoiem D (2015) Labeling complete surfaces in scene understanding. IJCV 112(2):172–187

    Article  MathSciNet  Google Scholar 

  12. Heitz G, Gal C (2010) Object separation in x-ray image sets. In: Proceedings of CVPR, pp 2093–2100

  13. Hinz S, Baumgartner A (2003) Automatic extraction of urban road networks from multi-view aerial imagery. ISPRS J Photogramm Remote Sens 58:83–98

    Article  Google Scholar 

  14. Hoiem D, Efros A, Hebert M (2005) Geometric context from a single image. In: Proceedings of ICCV, pp 654–661

  15. Kim BS, Kohli P, Savarese S (2013) 3D scene understanding by voxel-CRF. In: Proceedings of ICCV, pp 1425–1432

  16. Kolmogorov V (2006) Convergent tree-reweighted message passing for energy minimization. IEEE Trans Pattern Anal Mach Intell 28(10):1568–1583

    Article  Google Scholar 

  17. Kosov S (2015) Direct graphical models C++ library. http://research.project-10.de/dgm/

  18. Kosov S, Kohli P, Rottensteiner F, Heipke C (2013) A two-layer conditional random field for the classification of partially occluded objects. arXiv:1307.3043 [cs.CV]

  19. Kosov S, Rottensteiner F, Heipke C, Leitloff J, Hinz S. (2012) 3D classification of crossroads from multiple aerial images using Markov random fields. In: Proceedings of ISPRS Congress, pp 479–484

  20. Kramer O (2010) Iterated local search with Powell’s method: a memetic algorithm for continuous global optimization. Memetic Comput 2(1):69–83

    Article  Google Scholar 

  21. Kulkarni VY, Sinha PK (2013) Random forest classifiers: a survey and future research directions. Int J Adv Comput 36(1):1144–1153

    Google Scholar 

  22. Kumar S, Hebert M (2005) A hierarchical field framework for unified context-based classification. In: Proceedings of ICCV, pp 1284–1291

  23. Lafferty J, McCallum A, Pereira F (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of ICML, pp 282–289

  24. Lauritzen S, Wermuth N (1989) Graphical models for associations between variables, some of which are qualitative and some quantitative. Ann Stat 17(1):31–57

    Article  MathSciNet  Google Scholar 

  25. Leibe B, Leonardis A, Schiele B (2008) Robust object detection with interleaved categorization and segmentation. IJCV 77:259–289

    Article  Google Scholar 

  26. Li SZ (2009) Markov random field modeling in image analysis, 3rd edn. Springer, Berlin

    MATH  Google Scholar 

  27. Liu C, Kohli P, Furukawa Y (2016) Layered scene decomposition via the occlusion-CRF. In: Proceedings of CVPR, pp 165–173

  28. Liu C, Yuen J, Torralba A (2011) Nonparametric scene parsing via label transfer. IEEE Trans Pattern Anal Mach Intell 33(12):2368–2382

    Article  Google Scholar 

  29. Prasad M, Zisserman A, Fitzgibbon A, Kumar P, Torr P (2006) Learning class-specific edges for object detection and segmentation. In: Proceedings of ICVGIP, pp 94–105

  30. Richardson T, Spirtes P (2002) Ancestral graph Markov models. Ann Stat 30 (4):962–1030

    Article  MathSciNet  Google Scholar 

  31. Schindler K (2012) An overview and comparison of smooth labeling methods for land-cover classification. IEEE Trans Geosci Remote Sens:4534–4545

  32. Schnitzspan P, Fritz M, Roth S, Schiele B (2009) Discriminative structure learning of hierarchical representations for object detection. In: Proceedings of CVPR, pp 2238–2245

  33. Silberman N, Shapira L, Gal R, Kohli P (2014) A contour completion model for augmenting surface reconstructions. In: Proceedings of ECCV, pp 488–503

  34. Souly N, Shah M (2016) Scene labeling using sparse precision matrix. In: Proceedings of CVPR. IEEE Computer Society, pp 3650–3658

  35. Souly N, Spampinato C, Shah M (2017) Semi supervised semantic segmentation using generative adversarial network. In: Proceedings of ICCV, pp 5689–5697

  36. Szeliski R, Zabih R, Scharstein D, Veksler O, Kolmogorov V, Agarwala A, Tappen M, Rother C (2008) A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. IEEE Trans Pattern Anal Mach Intell 30(6):1068–1080

    Article  Google Scholar 

  37. Tighe J, Niethammer M, Lazebnik S (2014) Scene parsing with object instances and occlusion ordering. In: Proceedings of CVPR, pp 3748–3755

  38. Tu Z, Bai X (2010) Auto-context and its application to high-level vision tasks and 3D brain image segmentation. IEEE Trans Pattern Anal Mach Intell 32(10):1744–1757

    Article  Google Scholar 

  39. Vishwanathan S, Schraudolph N, Schmidt M, Murphy K (2006) Accelerated training of conditional random fields with stochastic gradient methods. In: Proceedings of ICML, pp 969–976

  40. Winn J, Shotton J (2006) The layout consistent random field for recognizing and segmenting partially occluded objects. In: Proceedings of CVPR, pp 37–44

  41. Yang Y, Hallman S, Ramanan D, Fowlkes C (2012) Layered object models for image segmentation. IEEE Trans Pattern Anal Mach Intell 34:1731–1743

    Article  Google Scholar 

  42. Yin Z, Collins R (2007) Belief propagation in a 3D spatiotemporal MRF for moving object detection. In: Proceedings of CVPR, pp 1–8

Download references

Author information

Authors and Affiliations

  1. Research Group for Pattern Recognition, Department ETI, University of Siegen, Hoelderlinstr. 3, 57076, Siegen, Germany

    Sergey Kosov, Kimiaki Shirahama & Marcin Grzegorzek

Authors
  1. Sergey Kosov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kimiaki Shirahama
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marcin Grzegorzek
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sergey Kosov.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kosov, S., Shirahama, K. & Grzegorzek, M. Labeling of partially occluded regions via the multi-layer CRF. Multimed Tools Appl 78, 2551–2569 (2019). https://doi.org/10.1007/s11042-018-6298-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6298-5

Keywords

Search

Navigation