Skip to main content
SpringerLink
Log in

An FPGA-based real-time occlusion robust stereo vision system using semi-global matching

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Stereo matching approaches are an appealing choice for acquiring depth information in a number of video processing applications. It is desirable that these solutions generate dense, robust disparity maps in real time. However, occlusion regions may disturb the applications that need these maps. Among the best of these approaches is the semi-global matching (SGM) technique. This paper presents an FPGA-based stereo vision system based on SGM. This system calculates disparity maps by streaming, which are scalable to several resolutions and disparity ranges. To increase the robustness of the SGM technique even further, the present work has implemented a combination of the gradient filter and the sampling-insensitive absolute difference in the pre-processing phase. Furthermore, as a post-processing step, this paper proposes a novel streaming architecture to detect noisy and occluded regions. The FPGA-based implementations of the proposed stereo matching system in two distinct heterogeneous architecture (GPP—general purpose processor, and FPGA) were evaluated using the Middlebury stereo vision benchmark. The achieved results reported a frame rate of 25 FPS for the disparity maps processing in HD resolution (1024 \(\times\) 768 pixels), with 256 disparity levels. The results have demonstrated that the memory utilization, processing performance, and accuracy are among the best of FPGA-based stereo vision systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Prisacariu, V.A., Kähler, O., Golodetz, S., Sapienza, M., Cavallari, T., Torr, P.H.S., Murray, D.W..: InfiniTAM v3: A Framework for Large-Scale 3D Reconstruction with Loop Closure. arXiv:1708.00783 (arXiv preprint) (2017)

  2. Keller, C.G., Enzweiler, M., Rohrbach, M., Llorca, D.F., Schnorr, C., Gavrila, D.M.: The benefits of dense stereo for pedestrian detection. IEEE Trans. Intell. Transport. Syst. 12(4), 1096–1106 (2011)

    Article  Google Scholar 

  3. Oleynikova, H., Honegger, D., Pollefeys, M.: Reactive avoidance using embedded stereo vision for MAV flight. In: Robotics and Automation (ICRA), 2015 IEEE International Conference on, pp. 50–56. IEEE (2015)

  4. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1), 7–42 (2002)

    Article  Google Scholar 

  5. Hirschmüller, H., Gehrig, S.: Stereo matching in the presence of sub-pixel calibration errors. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 437–444. IEEE (2009)

  6. Sun, J., Li, Y., Kang, S.B., Shum, H.Y.: Symmetric stereo matching for occlusion handling. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 2, pp. 399–406. IEEE (2005)

  7. Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008)

    Article  Google Scholar 

  8. Gehrig, S.K., Rabe, C.: Real-time semi-global matching on the CPU. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 85–92 (2010)

  9. Spangenberg, R., Langner, T., Adfeldt, S., Rojas, R.: Large scale semi-global matching on the CPU. In: 2014 IEEE Intelligent Vehicles Symposium Proceedings, pp. 195–201 (2014)

  10. Hernandez-Juarez, D., Chacón, A., Espinosa, A., Vázquez, D., Moure, J.C., López, A.M.: Embedded real-time stereo estimation via Semi-Global Matching on the GPU. Proced. Comput. Sci. 80, 143–153 (2016)

    Article  Google Scholar 

  11. Bailey, D.G.: Design for Embedded Image Processing on FPGAs, 1st edn. Wiley, Oxford (2011)

    Book  Google Scholar 

  12. Cambuim, L.F.S., Barbosa, J.P.F., Barros, E.N.S.: Hardware module for low-resource and real-time stereo vision engine using semi-global matching approach. In: Proceedings of the 30th Symposium on Integrated Circuits and Systems Design: Chip on the Sands, SBCCI ’17, pp. 53–58, New York, NY, USA. ACM (2017)

  13. Birchfield, S., Tomasi, C.: A pixel dissimilarity measure that is insensitive to image sampling. IEEE Trans. Pattern Anal. Mach. Intell. 20(4), 401–406 (1998)

    Article  Google Scholar 

  14. Honegger, D., Oleynikova, H., Pollefeys, M.: Real-time and low latency embedded computer vision hardware based on a combination of FPGA and mobile CPU. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4930–4935 (2014)

  15. Gehrig, S.K., Eberli, F., Meyer, T.: A real-time low-power stereo vision engine using semi-global matching. In: International Conference on Computer Vision Systems, pp. 134–143. Springer (2009)

  16. Banz, C., Hesselbarth, S., Flatt, H., Blume, H., Pirsch, P.: Real-time stereo vision system using semi-global matching disparity estimation: architecture and FPGA-implementation. In: Embedded Computer Systems (SAMOS), 2010 International Conference on, pp. 93–101. IEEE (2010)

  17. Wang, W., Yan, J., Xu, N., Wang, Y., Hsu, F.H.: Real-time high-quality stereo vision system in FPGA. In: 2013 International Conference on Field-Programmable Technology (FPT), pp. 358–361 (2013)

  18. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, New York (2003)

    MATH  Google Scholar 

  19. Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd edn. Prentice-Hall Inc, Upper Saddle River (2006)

    Google Scholar 

  20. Pantilie, C.D., Nedevschi, S.: SORT-SGM: subpixel optimized real-time semiglobal matching for intelligent vehicles. IEEE Trans. Veh. Technol. 61(3), 1032–1042 (2012)

    Article  Google Scholar 

  21. Benedetti, L., Corsini, M., Cignoni, P., Callieri, M., Scopigno, R.: Color to gray conversions in the context of stereo matching algorithms. Mach. Vis. Appl. 23(2), 327–348 (2012)

    Article  Google Scholar 

  22. Hirschmuller, H., Scharstein, D.: Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell. 31(9), 1582–1599 (2009)

    Article  Google Scholar 

  23. Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., Westling, P.: High-resolution stereo datasets with subpixel-accurate ground truth. In: German Conference on Pattern Recognition, pp. 31–42. Springer (2014)

  24. Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, vol. 1, pp I–I. IEEE (2003)

  25. Scharstein, D., Pal, C.: Learning conditional random fields for stereo. In: Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pp. 1–8. IEEE (2007)

  26. Hirschmüller, H., Scharstein, D.: Evaluation of cost functions for stereo matching. In: Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pp. 1–8. IEEE (2007)

  27. Wang, W., Yan, J., Xu, N., Wang, Y., Hsu, F.H.: Real-time high-quality stereo vision system in FPGA. IEEE Trans. Circ. Syst. Video Technol. 25(10), 1696–1708 (2015)

    Article  Google Scholar 

  28. Rahnama, O., Cavallari, T., Golodetz, S., Walker, S., Torr, P.H.S.: R3SGM: Real-time Raster-Respecting Semi-Global Matching for Power-Constrained Systems. arXiv:1810.12988 (arXiv preprint) (2018)

  29. Perri, S., Frustaci, F., Spagnolo, F., Corsonello, P.: Stereo vision architecture for heterogeneous systems-on-chip. J. Real Time Image Process. (2018)

  30. Schönberger, J.L., Sinha, S.N., Pollefeys, M.: Learning to fuse proposals from multiple scanline optimizations in semi-global matching. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 739–755 (2018)

  31. Žbontar, J., LeCun, Y.: Stereo matching by training a convolutional neural network to compare image patches. J. Mach. Learn. Res. 17(1), 2287–2318 (2016)

    MATH  Google Scholar 

  32. Sinha, S.N., Scharstein, D., Szeliski, R.: Efficient high-resolution stereo matching using local plane sweeps. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1582–1589 (2014)

  33. Chang, Q., Maruyama, T.: Real-time stereo vision system: a multi-block matching on GPU. IEEE Access 6, 42030–42046 (2018)

    Article  Google Scholar 

  34. Geiger, A., Roser, M., Urtasun, R.: Efficient large-scale stereo matching. In :Computer Vision–ACCV 2010, pp. 25–38. Springer (2010)

  35. Lee, J., Jun, D., Eem, C., Hong, H.: Improved census transform for noise robust stereo matching. Opt. Eng. 55(6), 063107 (2016)

    Article  Google Scholar 

  36. Yann, L., Yoshua, B.: Convolutional networks for images, speech, and time series. Handb. Brain Theory Neural Netw. 3361(10), 1995 (1995)

    Google Scholar 

  37. Matthew, J., Dustin, R., Matthew, H., Ryan, K.: RIFFA 2.1: a reusable integration framework for FPGA accelerators. ACM Trans. Reconfigur. Technol. Syst. 8(4), 22 (2015)

    Google Scholar 

  38. Mei, X., Sun, X., Zhou, M., Jiao, S., Wang, H., Zhang, X.: On building an accurate stereo matching system on graphics hardware. In: Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pp. 467–474. IEEE (2011)

  39. Robert, S., Tobias, ., Raúl, R.: Weighted semi-global matching and center-symmetric census transform for robust driver assistance. In: International Conference on Computer Analysis of Images and Patterns, pp. 34–41. Springer (2013)

Download references

Author information

Authors and Affiliations

  1. Informatics Center, Federal University of Pernambuco, Recife, Pernambuco, Brazil

    Lucas F. S. Cambuim, Luiz A. Oliveira Jr., Edna N. S. Barros & Antonyus P. A. Ferreira

Authors
  1. Lucas F. S. Cambuim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luiz A. Oliveira Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Edna N. S. Barros
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Antonyus P. A. Ferreira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lucas F. S. Cambuim.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cambuim, L.F.S., Oliveira, L.A., Barros, E.N.S. et al. An FPGA-based real-time occlusion robust stereo vision system using semi-global matching. J Real-Time Image Proc 17, 1447–1468 (2020). https://doi.org/10.1007/s11554-019-00902-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-019-00902-w

Keywords

Search

Navigation