Skip to main content
SpringerLink
Log in

3D entity-based stereo matching with ground control points and joint second-order smoothness prior

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Disparity estimation for a scene with complex geometric characteristics such as slanted or highly curved surfaces is a basic and important issue in stereo matching. Traditional methods often use first-order smoothness priors that always lead to low-curvature frontal-parallel disparity maps. We propose a stereo framework that views the scene as a set of 3D entities with compact and smooth disparity distributions. The 3D entity-based representation enables some contributions to obtain a precise disparity estimation. A GCPs-plane constraint based on ground control points is used to strengthen the compact distributions of the disparities in each entity by restricting the scope of the disparity variance and reducing matching ambiguities in repetitive or low-texture areas. Furthermore, we have formulated a joint second-order smoothness prior, which combines a geometric weight with the derivative of disparity values. This prior encourages smooth disparity variations inside each entity and means that each entity is biased towards being a 3D planar surface. Segmentation is incorporated as soft constraint by effectively fusing the advantages of the image color gradient and GCPs-plane. This avoids blending of the foreground and background and retains only the disparity discontinuities from geometrically smooth regions with strong texture gradients. Our framework is formulated as a maximum a posteriori probability estimation problem that is optimized using the fusion-move approach. Evaluation results on the Middlebury benchmark show that the proposed method ranks second among the approximately \(152\) listed algorithms. In addition, it performs well in real-world scenes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Birchfield, S., Tomasi, C.: A pixel dissimilarity measure that is insensitive to image sampling. IEEE Trans. Pattern Anal. Mach. Intell. 20(4), 401–406 (1998)

    Article  Google Scholar 

  2. Bleyer, M., Gelautz, M.: A layered stereo matching algorithm using image segmentation and global visibility constraints. ISPRS J. Photogr. Remote Sens. 59(3), 128–150 (2005)

    Article  Google Scholar 

  3. Bleyer, M., Rhemann, C., Rother, C.: Patchmatch stereo-stereo matching with slanted support windows. BMVC 11, 1–11 (2011)

    Google Scholar 

  4. Bleyer, M., Rother, C., Kohli, P.: Surface stereo with soft segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1570–1577. IEEE (2010)

  5. Borisagar, V.H., Zaveri, M.A.: A novel segment-based stereo matching algorithm for disparity map generation. In: International Conference on Computer and Software Modeling, vol. 14 (2011)

  6. Boros, E., Hammer, P.L., Tavares, G.: Preprocessing of unconstrained quadratic binary Optimization. Technical Report RRR 10–2006, RUTCOR Research Report (2006)

  7. Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. Int. J. Comput. Vis. 59(2), 167–181 (2004)

    Article  Google Scholar 

  8. Hirschmuller, H., Scharstein, D.: Evaluation of cost functions for stereo matching. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR’07, pp. 1–8. IEEE (2007)

  9. Ishikawa, H.: Higher-order clique reduction in binary graph cut. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 2993–3000. IEEE (2009)

  10. Kim, J., Kolmogorov, V., Zabih, R.: Visual correspondence using energy minimization and mutual information. In: Proceedings of Ninth IEEE International Conference on Computer Vision, 2003, pp. 1033–1040. IEEE (2003)

  11. Kim, J.C., Lee, K.M., Choi, B.T., Lee, S.U.: A dense stereo matching using two-pass dynamic programming with generalized ground control points. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, vol. 2, pp. 1075–1082. IEEE (2005)

  12. Klaus, A., Sormann, M., Karner, K.: Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: 18th International Conference on Pattern Recognition, 2006. ICPR 2006, vol. 3, pp. 15–18. IEEE (2006)

  13. Kolmogorov, V., Rother, C.: Minimizing nonsubmodular functions with graph cuts-a review. IEEE Trans. Pattern Anal. Mach. Intell. 29(7), 1274–1279 (2007)

    Article  Google Scholar 

  14. Kolmogorov, V., Zabih, R.: Multi-camera scene reconstruction via graph cuts. In: Computer VisionECCV 2002, pp. 82–96. Springer, Berlin (2002)

  15. Kolmogorov, V., Zabin, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004)

    Article  Google Scholar 

  16. Lee, Z., Juang, J., Nguyen, T.: Local disparity estimation with three-moded cross census and advanced support weight. IEEE Trans. Multimed. 15(8), 1855–1864 (2013)

    Article  Google Scholar 

  17. Lempitsky, V., Rother, C., Blake, A.: Logcut-efficient graph cut optimization for Markov random fields. In: IEEE 11th International Conference on Computer Vision, 2007. ICCV 2007, pp. 1–8. IEEE (2007)

  18. Li, G., Zucker, S.W.: Differential geometric consistency extends stereo to curved surfaces. In: Computer Vision-ECCV 2006, pp. 44–57. Springer, Berlin (2006)

  19. Li, G., Zucker, S.W.: Differential geometric inference in surface stereo. IEEE Trans. Pattern Anal. Mach. Intell. 32(1), 72–86 (2010)

    Article  Google Scholar 

  20. Liu, Z., Han, Z., Ye, Q., Jiao, J.: A new segment-based algorithm for stereo matching. In: International Conference on Mechatronics and Automation, 2009. ICMA 2009, pp. 999–1003. IEEE (2009)

  21. Mei, X., Sun, X., Dong, W., Wang, H., Zhang, X.: Segment-tree based cost aggregation for stereo matching. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 313–320. IEEE (2013)

  22. Middleburybenchmark: http://vision.middlebury.edu/stereo/

  23. Ogale, A.S., Aloimonos, Y.: Stereo correspondence with slanted surfaces: critical implications of horizontal slant. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004, vol. 1, pp. I–568. IEEE (2004)

  24. Oh, J.D., Jay Kuo, C.C.: Robust stereo matching with improved graph and surface models and occlusion handling. J. Vis. Commun. Image Represent. 21(5), 404–415 (2010)

    Article  Google Scholar 

  25. Oh, J.D., Ma, S., Kuo, C.C.: Stereo matching via disparity estimation and surface modeling. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR’07, pp. 1–8. IEEE (2007)

  26. Rhemann, C., Hosni, A., Bleyer, M., Rother, C., Gelautz, M.: Fast cost-volume filtering for visual correspondence and beyond. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3017–3024. IEEE (2011)

  27. Rother, C., Kolmogorov, V., Lempitsky, V., Szummer, M.: Optimizing binary MRFS via extended roof duality. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR’07, pp. 1–8. IEEE (2007)

  28. Scharstein, D., Pal, C.: Learning conditional random fields for stereo. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR’07, pp. 1–8. IEEE (2007)

  29. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1–3), 7–42 (2002)

    Article  MATH  Google Scholar 

  30. Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: Proceedings of 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003, vol. 1, pp. I–195. IEEE (2003)

  31. Song, P., Wu, X., Wang, M.Y.: Volumetric stereo and silhouette fusion for image-based modeling. Vis. Comput. 26(12), 1435–1450 (2010)

    Article  MathSciNet  Google Scholar 

  32. Stefano, L.D., Marchionni, M., Mattoccia, S.: A fast area-based stereo matching algorithm. Image Vis. Comput. 22(12), 983–1005 (2004)

    Article  Google Scholar 

  33. Sun, J., Li, Y., Kang, S.B., Shum, H.Y.: Symmetric stereo matching for occlusion handling. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, vol. 2, pp. 399–406. IEEE (2005)

  34. Sun, X., Mei, X., Jiao, S., Zhou, M., Wang, H.: Stereo matching with reliable disparity propagation. In: 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), pp. 132–139. IEEE (2011)

  35. Tanimoto, M., Tehrani, M.P., Fujii, T., Yendo, T.: Free-viewpoint TV. IEEE Signal Process. Mag. 28(1), 67–76 (2011)

    Article  Google Scholar 

  36. Tao, H., Sawhney, H.S., Kumar, R.: A global matching framework for stereo computation. In: Proceedings of Eighth IEEE International Conference on Computer Vision, 2001. ICCV 2001, vol. 1, pp. 532–539. IEEE (2001)

  37. Terzopoulos, D.: Multilevel computational processes for visual surface reconstruction. Comput. Vis. Graph. Image Process. 24(1), 52–96 (1983)

    Article  Google Scholar 

  38. Wang, D., Lim, K.B.: A new segment-based stereo matching using graph cuts. In: 2010 3rd IEEE International Conference on Computer Science and Information Technology (ICCSIT), vol. 5, pp. 410–416. IEEE (2010)

  39. Wang, D., Lim, K.B.: Obtaining depth map from segment-based stereo matching using graph cuts. J. Vis. Commun. Image Represent. 22(4), 325–331 (2011)

    Article  MathSciNet  Google Scholar 

  40. Wang, L., Yang, R.: Global stereo matching leveraged by sparse ground control points. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3033–3040. IEEE (2011)

  41. Woodford, O., Torr, P., Reid, I., Fitzgibbon, A.: Global stereo reconstruction under second-order smoothness priors. IEEE Trans. Pattern Anal. Mach. Intell. 31(12), 2115–2128 (2009)

    Article  Google Scholar 

  42. Xu, Z., Ma, L., Kimachi, M., Suwa, M.: Efficient contrast invariant stereo correspondence using dynamic programming with vertical constraint. Vis. Comput. 24(1), 45–55 (2008)

    Article  MATH  Google Scholar 

  43. Yang, Q.: A non-local cost aggregation method for stereo matching. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1402–1409. IEEE (2012)

  44. Yang, Q., Wang, L., Yang, R., Stewénius, H., Nistér, D.: Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling. IEEE Trans. Pattern Anal. Mach. Intell. 31(3), 492–504 (2009)

  45. Yoon, K.J., Kweon, I.S.: Adaptive support-weight approach for correspondence search. IEEE Trans. Pattern Anal. Mach. Intell. 28(4), 650–656 (2006)

    Article  Google Scholar 

  46. Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23, 600–608 (2004)

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported and funded by the National Natural Science Foundation of China (No. 61300131), the National Key Technology Research and Development Program of China (No. 2013BAK03B07), the National High Technology Research and Development Program of China (863 Program) (No. 2013AA013902). The authors thank Dr. Meghan Stephens from National University of Ireland for her kind assistance on editing the manuscript.

Author information

Authors and Affiliations

  1. The Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, No.6 Kexueyuan South Road Zhongguancun, Haidian District, Beijing, 100190, China

    Jing Liu, Chunpeng Li, Feng Mei & Zhaoqi Wang

  2. University of Chinese Academy of Sciences, No.19A Yuquan Road, Beijing, 100049, China

    Jing Liu & Feng Mei

Authors
  1. Jing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chunpeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Feng Mei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhaoqi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, J., Li, C., Mei, F. et al. 3D entity-based stereo matching with ground control points and joint second-order smoothness prior. Vis Comput 31, 1253–1269 (2015). https://doi.org/10.1007/s00371-014-1009-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-014-1009-3

Keywords

Search

Navigation