Skip to main content
SpringerLink
Log in

High accuracy correspondence field estimation via MST based patch matching

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper presents an effective framework for correspondence field estimation. The core idea is to construct pixel-level and superpixel-level patch matching to achieve high accuracy estimation as well as fast speed computation. To this end, a hybrid edge-preserving supported weighting approach is first developed, which contributes to better performance on the pixel level, especially on those in the regions of fine structures. Then, a local Minimum Spanning Tree (MST) is constructed to describe regions and develop the adaptive smooth penalty weights, so that the over-patching in large textureless regions can be effectively avoided. In addition, the MST is further extended to handle occlusions in way of edge preserving strategy. Finally, all the above treatments are collected into an optimization model where the objective function is developed in terms of Markov Random Filed (MRF). In computation, a fast yet efficient iterative optimization strategy is developed. Our approach achieves favorable place on optical flow benchmark, which locates at the top two and top four for endpoint error and angular error evaluations among more than 130 approaches listed in the webpage.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Susstrunk S (2012) Slic superpixels compared to state-of-the-art superpixel methods. TPAMI 34(11):2274–2282

    Article  Google Scholar 

  2. Baker S, Scharstein D, Lewis JP, Roth S, Black MJ, Szeliski R (2011) A database and evaluation methodology for optical flow. Int J Comput Vis 92(1):1–31

    Article  Google Scholar 

  3. Bao L, Song Y, Yang Q, Yuan H, Wang G (2014) Tree filtering: efficient structure-preserving smoothing with a minimum spanning tree. TIP 23(2):555–569

    MathSciNet  MATH  Google Scholar 

  4. Bao L, Yang Q, Jin H (2014) Fast edge-preserving patchmatch for large displacement optical flow. IEEE TIP

  5. Barnes C, Adviser-Finkelstein A (2011) Patchmatch: a fast randomized matching algorithm with application to image and video. Princeton University

  6. Barnes C, Shechtman E, Finkelstein A, Goldman D (2009) Patchmatch: a randomized correspondence algorithm for structural image editing. TOG 28(3):24

    Article  Google Scholar 

  7. Besse F, Rother C, Fitzgibbon A, Kautz J (2013) Pmbp: patchmatch belief propagation for correspondence field estimation. IJCV, 1–12

  8. Bleyer M, Rhemann C, Rother C (2011) Patchmatch stereo-stereo matching with slanted support windows. In: BMVC, vol 11, pp 1–11

  9. Chen Z, Jin H, Lin Z, Cohen S, Wu Y (2013) Large displacement optical flow from nearest neighbor fields. In: 2013 IEEE conference on computer vision and pattern recognition, pp 2443–2450

  10. Chen Z, Jin H, Lin Z, Cohen S, Wu Y (2013) Large displacement optical flow from nearest neighbor fields. In: CVPR. IEEE, pp 2443–2450

  11. Dai L, Zhang F, Mei X, Zhang X (2015) Fast minimax path-based joint depth interpolation. IEEE Signal Process Lett 22(5):623–627

    Article  Google Scholar 

  12. Drulea M, Nedevschi S (2013) Motion estimation using the correlation transform. TIP 22(8):3260–3270

    Google Scholar 

  13. Gupta RK, Cho SY (2010) Real-time stereo matching using adaptive binary window. 3DPVT 2:1–8

    Google Scholar 

  14. Hornáček M, Besse F, Kautz J, Fitzgibbon A, Rother C (2014) Highly overparameterized optical flow using patchmatch belief propagation. In: ECCV. Springer, pp 220–234

  15. Hosni A, Bleyer M, Gelautz M, Rhemann C (2009) Local stereo matching using geodesic support weights. In: ICIP. IEEE, pp 2093–2096

  16. Kim TH, Lee HS, Lee KM (2013) Optical flow via locally adaptive fusion of complementary data costs. In: ICCV. IEEE, pp 3344–3351

  17. Lempitsky V, Rother C, Roth S, Blake A (2010) Fusion moves for Markov random field optimization. TPAMI 32(8):1392–1405

    Article  Google Scholar 

  18. Lu J, Yang H, Min D, Do MN (2013) Patch match filter: efficient edge-aware filtering meets randomized search for fast correspondence field estimation. In: CVPR. IEEE, pp 1854–1861

  19. Mohamed MA, Mertsching B (2012) Tv-l1 optical flow estimation with image details recovering based on modified census transform. In: Advances in visual computing. Springer, pp 482–491

  20. Olsson C, Ulén J, Boykov Y (2013) In defense of 3d-label stereo. In: CVPR. IEEE, pp 1730–1737

  21. Scharstein D, Szeliski R (2002) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1–3):7–42

    Article  Google Scholar 

  22. Shen S (2013) Accurate multiple view 3d reconstruction using patch-based stereo for large-scale scenes. TIP 22(5):1901–1914

    MathSciNet  MATH  Google Scholar 

  23. Szeliski R, Zabih R, Scharstein D, Veksler O, Kolmogorov V, Agarwala A, Tappen M, Rother C (2008) A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. TPAMI 30(6):1068–1080

    Article  Google Scholar 

  24. Taniai T, Matsushita Y, Naemura T (2014) Graph cut based continuous stereo matching using locally shared labels. In: CVPR. IEEE, pp 1613–1620

  25. Tombari F, Mattoccia S, Di Stefano L (2007) Segmentation-based adaptive support for accurate stereo correspondence. In: Advances in image and video technology. Springer, pp 427–438

  26. Xu L, Jia J, Matsushita Y (2010) Motion detail preserving optical flow estimation. In: CVPR, pp 1293–1300

  27. Yang Q (2012) A non-local cost aggregation method for stereo matching. In: CVPR. IEEE, pp 1402–1409

  28. Yoon KJ, Kweon IS (2006) Adaptive support-weight approach for correspondence search. TPAMI 28(4):650–656

    Article  Google Scholar 

Download references

Acknowledgements

This work was also supported by the National Key R&D Program of China (Grant 2018YFB2100602). This work is supported by the National Natural Science Foundation of China (Nos. 61620106003, 61971418, 61771026, 61671451 and 61571046).

Author information

Author notes

    Authors and Affiliations

    1. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

      Feihu Zhang, Shibiao Xu & Xiaopeng Zhang

    Authors
    1. Feihu Zhang
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Shibiao Xu
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Xiaopeng Zhang
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding authors

    Correspondence to Shibiao Xu or Xiaopeng Zhang.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Feihu Zhang and Shibiao Xu contributed equally to this work and share the first authorship.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Zhang, F., Xu, S. & Zhang, X. High accuracy correspondence field estimation via MST based patch matching. Multimed Tools Appl 79, 13291–13309 (2020). https://doi.org/10.1007/s11042-020-08633-y

    Download citation

    • Received:

    • Revised:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11042-020-08633-y

    Keywords

    Search

    Navigation