Skip to main content
SpringerLink
Log in

Dynamic programming with adaptive and self-adjusting penalty for real-time accurate stereo matching

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Dense disparity map extraction is one of the most active research areas in computer vision. It tries to recover three-dimensional information from a stereo image pair. A large variety of algorithms has been developed to solve stereo matching problems. This paper proposes a new stereo matching algorithm, capable of generating the disparity map in real-time and with high accuracy. A novel stereo matching approach is based on per-pixel difference adjustment for the absolute differences, gradient matching and rank transform. The selected cost metrics are aggregated using guided filter. The disparity calculation is performed using dynamic programming with self-adjusting and adaptive penalties to improve disparity map accuracy. Our approach exploits mean-shift image segmentation and refinement technique to reach higher accuracy. In addition, a parallel high-performance graphics hardware based on Compute Unified Device Architecture is used to implement this method. Our algorithm runs at 36 frames per second on \(640 \times 480\) video with 64 disparity levels. Over 707 million disparity evaluations per second (MDE/s) are achieved in our current implementation. In terms of accuracy and runtime, our algorithm ranks the third place on Middlebury stereo benchmark in quarter resolution up to the submitting.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Hamzah, R.A., Ibrahim, H.: Literature survey on stereo vision disparity map algorithms. J. Sens. 2016, 1–23 (2016)

    Article  Google Scholar 

  2. Hirschmuller, H., Scharstein, D.: Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach Intell. 31(9), 1582–1599 (2009)

    Article  Google Scholar 

  3. Lu, H., Xu, H., Zhang, L., Ma, Y., Zhao, Y.: Cascaded multi-scale and multi-dimension convolutional neural network for stereo matching. In: IEEE visual communications and image processing (VCIP) (2018)

  4. Zhan, Y., Gu, Y., Huang, K., Zhang, C., Hu, K.: Accurate image-guided stereo matching with efficient matching cost and disparity refinement. IEEE Trans. Circuits Syst. Video Technol. 26(9), 1632–1645 (2016)

    Article  Google Scholar 

  5. Tan, P., Monasse, P.: Stereo disparity through cost aggregation with guided filter. Image Process Line 4, 252–275 (2014)

    Article  Google Scholar 

  6. Hamzah, R.A., Ibrahim, H., Hassan, A.H.A.: Stereo matching algorithm based on per pixel difference adjustment, iterative guided filter and graph segmentation. J. Vis. Commun. Image Represent. 42, 145–160 (2017)

    Article  Google Scholar 

  7. Rhemann, C., Hosni, A., Bleyer, M., Rother, C., Gelautz, M.: Fast cost-volume filtering for visual correspondence and beyond. In: IEEE Computer Vision and Pattern Recognition (CVPR) (2011)

  8. Hosni, A., Bleyer, M., Rhemann, C., Gelautz, M., Rother, C.: Real-time local stereo matching using guided image filtering. In: Proceedings of IEEE International Conference on Multimedia and Expo, pp. 1–6. Barcelona (2011)

  9. Hallek, M., Smach, F., Atri, M.: Real-time stereo matching on CUDA using Fourier descriptors and dynamic programming. Comput. Vis. Media 5(1), 59–71 (2019)

    Article  Google Scholar 

  10. Wang, L., Yang, R.G., Gong, M.L., Liao, M.: Real-time stereo using approximated joint bilateral filtering and dynamic programming. J. Real Time Image Process. 9(3), 447–461 (2014)

    Article  Google Scholar 

  11. Congote, J., Barandiaran,J., Barandiaran, I., Ruiz, O.: Realtime dense stereo matching with dynamic programming in CUDA. In: Proceedings of the 19th Spanish Congress of Graphical Informatics, pp. 231–234 (2009)

  12. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002)

    Article  Google Scholar 

  13. Yoon, K.J., Kweon, I.S.: Adaptive support-weight approach for correspondence search. IEEE Trans. Pattern Anal. Mach. Intell. 28(4), 650–656 (2006)

    Article  Google Scholar 

  14. Hirschmuller, H., Innocent, P.R., Garibaldi, J.: Real-time correlation-based stereo vision with reduced border errors. Int. J. Comput. Vis. 47, 229–246 (2002)

    Article  Google Scholar 

  15. Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: Proceedings of the Third European Conference on Computer Vision, Stockholm, pp. 151–158 (1994)

  16. Mei, X., Sun, X., Zhou, M., Jiao, S., Wang, H., Zhang, X.: On building an accurate stereo matching system on graphics hardware. In: GPUCV, Barcelona (2011)

  17. Kordelas, G., Alexiadis, D., Daras, P., Izquierdo, E.: Enhanced disparity estimation in stereo images. Image Vis. Comput. 35, 31–49 (2015)

    Article  Google Scholar 

  18. Kowalczuk, J., Psota, E.T., Perez, L.C.: Real-time stereo matching on CUDA using an iterative refinement method for adaptive support-weight correspondences. IEEE Trans. Circuits Syst. Video Technol. 23(1), 94–104 (2013)

    Article  Google Scholar 

  19. Yang, Q.: A non-local cost aggregation method for stereo matching. In: IEEE Computer Vision and Pattern Recognition, pp. 1402-1409 (2012)

  20. Zhu, S., Wang, Z., Zhang, X., Li, Y.: Edge-preserving guided filtering based cost aggregation for stereo matching. J. Vis. Commun. Image Represent. 39, 107–119 (2016)

    Article  Google Scholar 

  21. Mattoccia, S., Viti, M., Ries, F.: Near real-time fast bilateral stereo on the GPU. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–143 (2011)

  22. Wu, W., Zhu, H., Yu, S., Shi, J.: Stereo matching with fusing adaptive support weights. IEEE Access 7, 61960–61974 (2019)

    Article  Google Scholar 

  23. Yang, Q., Ji, P., Li, D., Yao, S., Zhang, M.: Fast stereo matching using adaptive guided filtering. Image Vis. Comput. 32, 202–211 (2014)

    Article  Google Scholar 

  24. Veksler, O.: Stereo correspondence by dynamic programming on a tree. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2005)

  25. Stentoumis, C., Karkalou, E., Karras, G.: A review and evaluation of penalty functions for semi-global matching. In Proc. IEEE Int. Conf. Intelligent Computer Communication Processing, pp. 167–172 (2015)

  26. Karkalou, E., Stentoumis, C., Karras, G.: Semi-global matching with self-adjusting penalties. Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 12, 353–360 (2017)

    Article  Google Scholar 

  27. Hirschmüller, H.: Stereo processing by semi-global matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008)

    Article  Google Scholar 

  28. Wang, W., Yan, J., Xu, N., Wang, Y., Hsu, F.H.: Real-time high quality stereo vision system in FPGA. IEEE Trans. Circ. Syst. Video Technol. 25(10), 1696–1708 (2015)

    Article  Google Scholar 

  29. Cambuim, L., Oliveira, L., Barros, E., Ferreira, A.: An FPGA based real-time occlusion robust stereo vision system using semiglobal matching. J. Real Time Image Proc. 30, 1–22 (2019)

    Google Scholar 

  30. Chang, Q., Maruyama, T.: Real-time stereo vision system: a multiblock matching on GPU. IEEE Access 6, 42030–42046 (2018)

    Article  Google Scholar 

  31. Wang, Z.F., Zheng, Z.G.: A region based stereo matching algorithm using cooperative optimization. In: IEEE Comput. Soc. Conf. Comput. Vision Pattern Recogn, pp. 1–8 (2008)

  32. Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24, 603–619 (2002)

    Article  Google Scholar 

  33. Wang, L., Liao, M., Gong, M., Yang, R., Nister, D.: High-quality real-time stereo using adaptive cost aggregation and dynamic programming. In: Proceedings of International Symposium on 3D Data Processing, Visualization and Transmission (2006)

  34. Ma, Z., He, K., Wei, Y., Sun, J., Wu, E.: Constant time weighted median filtering for stereo matching and beyond. In: Proceedings of International Conference on Computer Vision (2013)

  35. Zhang, Q., Xu, L., Jia, J.: 100+ times faster weighted median filter. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2014)

  36. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013)

    Article  Google Scholar 

  37. Salmen, J., Schlipsing, M., Edelbrunner, J., Hegemann, S., Lueke, S.: Real-time stereo vision: making more out of dynamic programming. In: Proceedings of International Conference on Computer Analysis of Images and Patterns (2009)

  38. Mozerov, M.G., van de Weijer, J.: Accurate stereo matching by two-step energy minimization. IEEE Trans. Image Process. 24(3), 1153–1163 (2015)

    Article  MathSciNet  Google Scholar 

  39. Jiao, J., Wang, R., Wang, W., Dong, S., Wang, Z., Gao, W.: Local stereo matching with improved matching cost and disparity refinement. IEEE MultiMed. 21(4), 16–27 (2014)

    Article  Google Scholar 

  40. Ait-Jellal, R., Lange, M., Wassermann, B., Schilling, A., Zell, A.: LS-ELAS: line segment based efficient large scale stereo matching. In: Proc. IEEE International Conference on Robotics and Automation (2017)

  41. Yin, J., Zhu, H., Yuan, D., Xue, T.: Sparse representation over discriminative dictionary for stereo matching. Pattern Recogn. 71, 278–289 (2017)

    Article  Google Scholar 

  42. Hamzah, R.A., Kadmin, A.F., Hamid, M.S., Ghani, S.F.A., Ibrahim, H.: Improvement of stereo matching algorithm for 3D surface reconstruction. Signal Process. Image Commun. 65, 165–172 (2018)

    Article  Google Scholar 

  43. Boitumelo, R., Jonas, M., Martin, W., Stefan, H., Jurgen, B.: ReS2tAC-UAV-borne real-time SGM stereo optimized for embedded ARM and CUDA devices. MDPI Sensors 21(11) (2021)

  44. Wang, Q., Shi, S., Zheng, S., Zhao, K., Chu, X.: Fadnet: a fast and accurate network for disparity estimation. In: IEEE international conference on robotics and automation (ICRA), pp. 101–107 (2020)

  45. Chang, Q., Maruyama, T.: Real-Time High-Quality Stereo Matching System on a GPU. In: IEEE 29th International Conference on Application-specific Systems, Architectures and Processors, pp. 1–8 (2018)

Download references

Acknowledgements

The authors extend their appreciation to the Deanship of Scientific Research at King Khalid University for funding this work through research groups program under grant number GRP/337/42

Author information

Authors and Affiliations

  1. Faculty of Sciences of Monastir, Monastir, Tunisia

    Mohamed Hallek & Hamdi Boukamcha

  2. National Engineering School of Monastir, Monastir, Tunisia

    Abdellatif Mtibaa

  3. College of Computer Science, King Khalid University, Abha, Saudi Arabia

    Mohamed Atri

Authors
  1. Mohamed Hallek
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hamdi Boukamcha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abdellatif Mtibaa
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohamed Atri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohamed Hallek.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hallek, M., Boukamcha, H., Mtibaa, A. et al. Dynamic programming with adaptive and self-adjusting penalty for real-time accurate stereo matching. J Real-Time Image Proc 19, 233–245 (2022). https://doi.org/10.1007/s11554-021-01180-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01180-1

Keywords

Search

Navigation