Skip to main content
SpringerLink
Log in

Joint view synthesis and disparity refinement for stereo matching

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Typical stereo algorithms treat disparity estimation and view synthesis as two sequential procedures. In this paper, we consider stereo matching and view synthesis as two complementary components, and present a novel iterative refinement model for joint view synthesis and disparity refinement. To achieve the mutual promotion between view synthesis and disparity refinement, we apply two key strategies, disparity maps fusion and disparity-assisted plane sweep-based rendering (DAPSR). On the one hand, the disparity maps fusion strategy is applied to generate disparity map from synthesized view and input views. This strategy is able to detect and counteract disparity errors caused by potential artifacts from synthesized view. On the other hand, the DAPSR is used for view synthesis and updating, and is able to weaken the interpolation errors caused by outliers in the disparity maps. Experiments on Middlebury benchmarks demonstrate that by introducing the synthesized view, disparity errors due to large occluded region and large baseline are eliminated effectively and the synthesis quality is greatly improved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bleyer M, Rother C, Kohli P. Surface stereo with soft segmentation. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. 2010, 1570–1577

    Google Scholar 

  2. Fickel G P, Jung C R, Malzbender T, Samadani R, Culbertson B. Stereo matching and view interpolation based on image domain triangulation. IEEE Transactions on Image Processing, 2013, 22(9): 3353–3365

    Article  MathSciNet  MATH  Google Scholar 

  3. Vogel C, Schindler K, Roth S. 3D scene flow estimation with a piecewise rigid scene model. International Journal of Computer Vision, 2015, 115(1): 1–28

    Article  MathSciNet  MATH  Google Scholar 

  4. Wu G, Masia B, Jarabo A, Zhang Y, Wang L, Dai Q, Chai T, Liu Y. Light field image processing: an overview. IEEE Journal of Selected Topics in Signal Processing, 2017, 11(7): 926–954

    Article  Google Scholar 

  5. Wu G, Liu Y, Fang L, Dai Q, Chai T. Light field reconstruction using deep convolutional network on EPI and extended applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, DOI:10.1109/TPAML.2018.2845393

  6. Fehn C. Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In: Proceedings of SPIE Stereoscopic Displays and Virtual Reality Systems XI. 2004, 93–104

    Chapter  Google Scholar 

  7. Hosni A, Rhemann C, Bleyer M, Rother C, Gelautz M. Fast costvolume filtering for visual correspondence and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(2): 504–511

    Article  Google Scholar 

  8. Lee S, Lee J H, Lim J, Suh I H. Robust stereo matching using adaptive random walk with restart algorithm. Image and Vision Computing, 2015, 37(C): 1–11

    Article  Google Scholar 

  9. Sun D, Roth S, Black M J. A quantitative analysis of current practices in optical flow estimation and the principles behind them. International Journal of Computer Vision, 2014, 106(2): 115–137

    Article  Google Scholar 

  10. Kubota A, Smolic A, Magnor M, Tanimoto M, Chen T, Zhang C. Multiview imaging and 3DTV. IEEE Signal Processing Magazine, 2007, 24(6): 10–21

    Article  Google Scholar 

  11. Stefanoski N, Wang O, Lang M, Greisen P, Heinzle S, Smolic A. Automatic view synthesis by image-domain-warping. IEEE Transactions on Image Processing, 2013, 22(9): 3329–3341

    Article  MathSciNet  MATH  Google Scholar 

  12. Tian D, Lai P L, Lopez P, Gomila C. View synthesis techniques for 3D video. In: Proceedings of Internatlbnal Society for Optics and Phontonics, Applications of Digital Image Processing XXXII. 2009, 7443: 74430T

    Google Scholar 

  13. Mori Y, Fukushima N, Yendo T, Fujii T, Tanimoto M. View generation with 3D warping using depth information for FTV. Signal Processing: Image Communication, 2009, 24(1–2): 65–72

    Google Scholar 

  14. Wanner S, Goldluecke B. Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(3): 606–619

    Article  Google Scholar 

  15. Zhang L, Zhang Y H, Huang H. Efficient variational light field view synthesis for making stereoscopic 3D images. Computer Graphics Forum. 2015, 7(34): 183–191

    Google Scholar 

  16. Zhang C, Li Z, Cheng Y, Cai R, Chao H, Rui Y. Meshstereo: a global stereo model with mesh alignment regularization for view interpolation. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 2057–2065

    Google Scholar 

  17. Kalantari N K, Wang T C, Ramamoorthi R. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics, 2016, 35(6): 193

    Article  Google Scholar 

  18. Scharstein D, Szeliski R. A taxonomy and evaluation of dense twoframe stereo correspondence algorithms. International Journal of Computer Vision, 2002, 47(1–3): 7–42

    Article  MATH  Google Scholar 

  19. Kordelas G A, Alexiadis D S, Daras P, Izquierdo E. Enhanced disparity estimation in stereo images. Image and Vision Computing, 2015, 35: 31–49

    Article  Google Scholar 

  20. Peng Y, Li G, Wang R, Wang W. Stereo matching with spaceconstrained cost aggregation and segmentation-based disparity refinement. In: Proceedings of International Society for Optics and Phoritonics, Three-Dimensional Image Processing, Measurement, and Applications. 2015, 9393: 939309

    Google Scholar 

  21. Zbontar J, LeCun Y. Stereo matching by training a convolutional neural network to compare image patches. Journal of Machine Learning Research, 2016, 17(1): 2287–2318

    MATH  Google Scholar 

  22. Luo W, Schwing A G, Urtasun R. Efficient deep learning for stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 5695–5703

    Google Scholar 

  23. Mattoccia S, Viti M, Ries F. Near real-time fast bilateral stereo on the GPU. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2011, 136–143

    Google Scholar 

  24. Mattoccia S, Giardino S, Gambini A. Accurate and efficient cost aggregation strategy for stereo correspondence based on approximated joint bilateral filtering. In: Proceedings of Asian Conference on Computer Vision. 2009, 371–380

    Google Scholar 

  25. Yoon K J, Kweon I S. Adaptive support-weight approach for correspondence search. IEEE Transactions on Pattern Analysis andMachine Intelligence, 2006, 28(4): 650–656

    Article  Google Scholar 

  26. Kolmogorov V, Zabih R. Computing visual correspondence with occlusions using graph cuts. In: Proceedings of the 8th IEEE International Conference on Computer Vision. 2001, 508–515

    Google Scholar 

  27. Taniai T, Matsushita Y, Naemura T. Graph cut based continuous stereo matching using locally shared labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1613–1620

    Google Scholar 

  28. Yang Q, Wang L, Yang R, Stewénius H, Nistér D. Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(3): 492–504

    Article  Google Scholar 

  29. Guney F, Geiger A. Displets: resolving stereo ambiguities using object knowledge. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 4165–4175

    Google Scholar 

  30. Psota E T, Kowalczuk J, Mittek M, Perez L C. Map disparity estimation using hidden markov trees. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 2219–2227

    Google Scholar 

  31. Mozerov M G, Van de W J. Accurate stereo matching by two-step energy minimization. IEEE Transactions on Image Processing, 2015, 24(3): 1153–1163

    Article  MathSciNet  MATH  Google Scholar 

  32. Mei X, Sun X, Zhou M, Jiao S, Wang H, Zhang X. On building an accurate stereo matching system on graphics hardware. In: Proceedings of the IEEE Conference on Computer Vision Workshops. 2011, 467–474

    Google Scholar 

  33. Veksler O. Fast variable window for stereo correspondence using integral images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2003, 556–561

    Google Scholar 

  34. Klaus A, Sormann M, Karner K. Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: Proceedings of the 18th International Conference on Pattern Recognition. 2006, 15–18

    Google Scholar 

  35. Vogel C, Schindler K, Roth S. 3D scene flow estimation with a rigid motion prior. In: Proceedings of the IEEE International Conference on Computer Vision. 2011, 1291–1298

    Google Scholar 

  36. Boykov Y, Veksler O, Zabih R. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(11): 1222–1239

    Article  Google Scholar 

  37. Yang Q, Yang R, Davis J, Nistér D. Spatial-depth super resolution for range images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2007, 1–8

    Google Scholar 

  38. Boominathan V, Mitra K, Veeraraghavan A. Improving resolution and depth-of-field of light field cameras using a hybrid imaging system. In: Proceedings of the IEEE International Conference on Computational Photography. 2014, 1–10

    Google Scholar 

  39. Ma Z, He K, Wei Y, Sun J, Wu E. Constant time weighted median filtering for stereo matching and beyond. In: Proceedings of the IEEE International Conference on Computer Vision. 2013, 49–56

    Google Scholar 

  40. Zhang Q, Xu L, Jia J. 100+ times faster weighted median filter (WMF). In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 2830–2837

    Google Scholar 

  41. Lin Z, Shum H Y. A geometric analysis of light field rendering. International Journal of Computer Vision, 2004, 58(2): 121–138

    Article  Google Scholar 

  42. Scharstein D, Szeliski R. High-accuracy stereo depth maps using structured light. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2003, 195–202

    Google Scholar 

  43. Scharstein D, Hirschmüller H, Kitajima Y, Krathwohl G, Nešić N, Wang X, Westling P. High-resolution stereo datasets with subpixelaccurate ground truth. In: Proceedings of German Conference on Pattern Recognition. 2014, 31–42

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National key foundation for exploring scientific instrument (2013YQ140517), the National Natural Science Foundation of China (Grant No. 61522111) and the Shenzhen Peacock Plan (KQTD20140630115140843).

Author information

Authors and Affiliations

  1. State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, Shenyang, 110819, China

    Gaochang Wu

  2. Broadband Network & Digital Media Lab, Department of Automation, Tsinghua University, Beijing, 100084, China

    Gaochang Wu, Yipeng Li & Yebin Liu

  3. Orbbec Company, Shenzhen, 518061, China

    Yuanhao Huang

Authors
  1. Gaochang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yipeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuanhao Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yebin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yebin Liu.

Additional information

Gaochang Wu received the BS and MS degrees in mechanical engineering from Northeastern University, China in 2013 and 2015, respectively. He is currently working toward PhD degree in control theory and control engineering in Northeastern University, China. His current research interests include data mining, signal analysis, image processing and computational photography.

Yipeng Li received the BS and MS degrees in electronic engineering from the Harbin Institute of Technology, China in 2003 and 2005, respectively and the PhD degree in electronic engineering from Tsinghua University, China in 2011. Since 2011, he has been a Research Associate with the Department of Automation, Tsinghua University. His current research interests include computer vision and data mining by using deep architecture networks.

Yuanhao Huang received the BS degree from the Peking University, China in 2002 and the PhD degree from the City University of Hong Kong, China in 2009. He was a research fellow in the Singapore- MIT Alliance for Research and Technology (SMART) in 2011. He is the expert of the Thousand Talents Plan (China), and the CEO of ORBBEC CO.,LTD. His research interests include optical measurement, computer vision and infrared technology.

Yebin Liu received the BE degree from Beijing University of Posts and Telecommunications, China in 2002 and the PhD degree from the Automation Department, Tsinghua University, China in 2009. He has been working as a research fellow at the computer graphics group of the Max Planck Institute for Informatik, Germany in 2010. He is currently an associate professor in Tsinghua University. His research areas include computer vision and computer graphics.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, G., Li, Y., Huang, Y. et al. Joint view synthesis and disparity refinement for stereo matching. Front. Comput. Sci. 13, 1337–1352 (2019). https://doi.org/10.1007/s11704-018-8099-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-018-8099-4

Keywords

Search

Navigation