Skip to main content
SpringerLink
Log in

Stereo Correspondence Evaluation Methods: A Systematic Review

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2015)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 9475))

Included in the following conference series:

Abstract

The stereo correspondence problem has received significant attention in literature during approximately three decades. During that period of time, the development on stereo matching algorithms has been quite considerable. In contrast, the proposals on evaluation methods for stereo matching algorithms are not so many. This is not trivial issue, since an objective assessment of algorithms is required not only to measure improvements on the area, but also to properly identify where the gaps really are, and consequently, guiding the research. In this paper, a systematic review on evaluation methods for stereo matching algorithms is presented. The contributions are not only on the found results, but also on how it is explained and presented: aiming to be useful for the researching community on visual computing, in which such systematic review process is not yet broadly adopted.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Morales, S., Klette, R.: A third eye for performance evaluation in stereo sequence analysis. In: Jiang, X., Petkov, N. (eds.) CAIP 2009. LNCS, vol. 5702, pp. 1078–1086. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  2. Keller, C.G., Enzweiler, M., Gavrila, D.M.: A new benchmark for stereo-based pedestrian detection. In: Intelligent Vehicles Symposium (IV), pp. 691–696 IEEE (2011)

    Google Scholar 

  3. Nielsen, M., Andersen, H.J., Slaughter, D.C., Granum, E.: Ground truth evaluation of computer vision based 3D reconstruction of synthesized and real plant images. Precis. Agric. 8, 49–62 (2007)

    Article  Google Scholar 

  4. Wang, Z.-F., Zheng, Z.-G.: A region based stereo matching algorithm using cooperative optimization. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008, pp. 1–8. IEEE (2008)

    Google Scholar 

  5. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002)

    Article  MATH  Google Scholar 

  6. Cabezas, I.: Evaluation of disparity maps, Doctoral thesis. Universidad del Valle (2013)

    Google Scholar 

  7. Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Proceedings. 2003, pp. I–195. IEEE (2003)

    Google Scholar 

  8. Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., Westling, P.: High-resolution stereo datasets with subpixel-accurate ground truth. In: Jiang, X., Hornegger, J., Koch, R. (eds.) GCPR 2014. LNCS, vol. 8753, pp. 31–42. Springer, Heidelberg (2014)

    Google Scholar 

  9. Cabezas, I., Padilla, V., Trujillo, M.: BMPRE: An error measure for evaluating disparity maps. In: ICSP, pp. 1051–1055 (2012)

    Google Scholar 

  10. Cabezas, I., Padilla, V., Trujillo, M.: A measure for accuracy disparity maps evaluation. In: San Martin, C., Kim, S.-W. (eds.) CIARP 2011. LNCS, vol. 7042, pp. 223–231. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  11. Haeusler, R., Klette, R.: Evaluation of stereo confidence measures on synthetic and recorded image data. In: International Conference on Informatics, Electronics & Vision (ICIEV), pp. 963–968. IEEE (2012)

    Google Scholar 

  12. Kitchenham, B.: Procedures for performing systematic reviews. Keele University, Keele, UK, pp. 1–26 (2004)

    Google Scholar 

  13. Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: EASE, pp. 68–77. British Computer Society, Italy (2008)

    Google Scholar 

  14. Morales, S., Klette, R.: Ground truth evaluation of stereo algorithms for real world applications. In: Koch, R., Huang, F. (eds.) ACCV 2010 Workshops, Part II. LNCS, vol. 6469, pp. 152–162. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  15. Hirschmuller, H., Scharstein, D.: Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Analy. Mach. Intell. 31, 1582–1599 (2009)

    Article  Google Scholar 

  16. Leclercq, P., Morris, J.: Robustness to noise of stereo matching. In: Proceedings of 12th International Conference on Image Analysis and Processing, pp. 606–611 IEEE (2003)

    Google Scholar 

  17. Szeliski, R., Zabih, R.: An experimental comparison of stereo algorithms. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) ICCV-WS 1999. LNCS, vol. 1883, pp. 1–19. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  18. Szeliski, R.: Prediction error as a quality metric for motion and stereo. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, pp. 781–788 IEEE (1999)

    Google Scholar 

  19. Haeusler, R., Klette, R.: Benchmarking stereo data (not the matching algorithms). In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.) Pattern Recognition. LNCS, vol. 6376, pp. 383–392. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  20. Sellent, A., Wingbermühle, J.: Quality assessment of non-dense image correspondences. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012 Ws/Demos, Part II. LNCS, vol. 7584, pp. 114–123. Springer, Heidelberg (2012)

    Google Scholar 

  21. Kondermann, D., Nair, R., Meister, S., Mischler, W., Güssefeld, B., Honauer, K., Hofmann, S., Brenner, C., Jähne, B.: Stereo ground truth with error bars. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9007, pp. 595–610. Springer, Heidelberg (2015)

    Google Scholar 

  22. Kostlivá, J., Čech, J.: Others: Feasibility boundary in dense and semi-dense stereo matching. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2007, pp. 1–8 IEEE (2007)

    Google Scholar 

  23. Neilson, D., Yang, Y.-H.: Evaluation of constructable match cost measures for stereo correspondence using cluster ranking. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1–8. IEEE (2008)

    Google Scholar 

  24. Malpica, W., Bovik, A.C.: Range image quality assessment by structural similarity. In: Furht, B. (ed.) Encyclopedia of Multimedia, pp. 757–762. Springer, New York (2008)

    Google Scholar 

  25. Shen, Y., Lu, C., Xu, P., Xu, L.: Objective quality assessment of noised stereoscopic images. In: ICMTMA, pp. 745–747. IEEE (2011)

    Google Scholar 

  26. van der Mark, W., Gavrila, D.M.: Real-time dense stereo for intelligent vehicles. IEEE Trans. Intell. Transp. Syst. 7, 38–50 (2006)

    Article  Google Scholar 

  27. Leibe, B., Cornelis, N., Cornelis, K., Van Gool, L.: Dynamic 3D scene analysis from a moving vehicle. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2007, pp. 1–8. IEEE (2007)

    Google Scholar 

  28. Steingrube, P., Gehrig, S.K., Franke, U.: Performance evaluation of stereo algorithms for automotive applications. In: Fritz, M., Schiele, B., Piater, J.H. (eds.) ICVS 2009. LNCS, vol. 5815, pp. 285–294. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  29. Morales, S., Vaudrey, T., Klette, R.: Robustness evaluation of stereo algorithms on long stereo sequences. In: Intelligent Vehicles Symposium, pp. 347–352. IEEE (2009)

    Google Scholar 

  30. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354–3361. IEEE (2012)

    Google Scholar 

  31. Hamilton, O.K., Breckon, T.P., Bai, X., Kamata, S.: A foreground object based quantitative assessment of dense stereo approaches for use in automotive environments. In: 20th IEEE International Conference on Image Processing (ICIP), pp. 418–422 IEEE (2013)

    Google Scholar 

  32. Woodward, A., Leclercq, P., Delmas, P., Gimel’farb, G.: Generation of an accurate facial ground truth for stereo algorithm evaluation. In: Wojciechowski, K., Smolka, B., Palus, H., Kozera, R.S., Skarbek, W., Noakes, L. (eds.) CVG 2006. Computational Imaging and Vision, pp. 534–539. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  33. Gong, M., Yang, R., Wang, L., Gong, M.: A performance study on different cost aggregation approaches used in real-time stereo matching. Int. J. Comput. Vis. 75, 283–296 (2007)

    Article  Google Scholar 

  34. Tombari, F., Mattoccia, S., Di Stefano, L.: Stereo for robots: quantitative evaluation of efficient and low-memory dense stereo algorithms. In: 11th International Conference on Control Automation Robotics & Vision (ICARCV), pp. 1231–1238. IEEE (2010)

    Google Scholar 

  35. Kelly, P.: Pedestrian detection and tracking using stereo vision techniques. Dublin City University (2007)

    Google Scholar 

  36. Kelly, P., O’Connor, N.E., Smeaton, A.F.: A framework for evaluating stereo-based pedestrian detection techniques. IEEE Trans. Circ. Syst. Video Technol. 18, 1163–1167 (2008)

    Article  Google Scholar 

  37. Kogler, J., Eibensteiner, F., Humenberger, M., Gelautz, M., Scharinger, J.: Ground truth evaluation for event-based silicon retina stereo data. In: CVPRW, pp. 649–656. IEEE (2013)

    Google Scholar 

  38. Aguilar, M.A., del Mar Saldana, M., Aguilar, F.J.: Generation and quality assessment of stereo-extracted DSM from GeoEye-1 and WorldView-2 imagery. IEEE Trans. Geosci. Remote Sens. 52, 1259–1271 (2014)

    Article  Google Scholar 

  39. Shin, B.-S., Caudillo, D., Klette, R.: Evaluation of two stereo matchers on long real-world video sequences. Pattern Recogn. 48, 1113–1124 (2015)

    Article  Google Scholar 

  40. Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., Cremers, D.: Efficient dense scene flow from sparse or dense stereo data. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 739–751. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  41. Fuhr, G., Fickel, G.P., Dal’Aqua, L.P., Jung, C.R., Malzbender, T., Samadani, R.: An evaluation of stereo matching methods for view interpolation. In: 20th IEEE International Conference on Image Processing (ICIP), pp. 403–407. IEEE (2013)

    Google Scholar 

  42. Vandewalle, P., Varekamp, C.: Disparity map quality for image-based rendering based on multiple metrics. In: International Conference on 3D Imaging (IC3D), pp. 1–5. IEEE (2014)

    Google Scholar 

  43. Hu, X., Mordohai, P.: A quantitative evaluation of confidence measures for stereo vision. IEEE Trans. Pattern Analy. Mach. Intell. 34, 2121–2133 (2012)

    Article  Google Scholar 

  44. Zhang, Z., Hou, C., Shen, L., Yang, J.: An objective evaluation for disparity map based on the disparity gradient and disparity acceleration. In: ITCS, pp. 452–455. IEEE (2009)

    Google Scholar 

  45. Varekamp, C., Hinnen, K., Simons, W.: Detection and correction of disparity estimation errors via supervised learning. In: 2013 International Conference on 3D Imaging (IC3D), pp. 1–7. IEEE (2013)

    Google Scholar 

  46. Haeusler, R., Nair, R., Kondermann, D.: Ensemble learning for confidence measures in stereo vision. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 305–312. IEEE (2013)

    Google Scholar 

  47. Milani, S., Ferrario, D., Tubaro, S.: No-reference quality metric for depth maps. In: 20th IEEE International Conference on Image Processing (ICIP), pp. 408–412. IEEE (2013)

    Google Scholar 

  48. Hermann, S., Morales, S., Klette, R.: Half-resolution semi-global stereo matching. In: Intelligent Vehicles Symposium (IV), pp. 201–206. IEEE (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Departamento de Ciencias de la Computación Y de la Decisión, Facultad de Minas, Universidad Nacional de Colombia., Bogotá, Colombia

    Camilo Vargas & John W. Branch

  2. Laboratorio de Investigación para el Desarrollo de la Ingeniería de Software, Universidad de San Buenaventura, Avenida 10 de Mayo, La Umbría, Vía a Pance, Cali, Colombia

    Ivan Cabezas

Authors
  1. Camilo Vargas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ivan Cabezas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John W. Branch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Camilo Vargas .

Editor information

Editors and Affiliations

  1. University of Nevada, Reno, Nevada, USA

    George Bebis

  2. NASA Ames Research Center, Moffett Field, California, USA

    Richard Boyle

  3. Lawrence Berkeley National Laboratory, Berkeley, California, USA

    Bahram Parvin

  4. Desert Research Institute, Reno, Nevada, USA

    Darko Koracin

  5. University of Houston, Houston, Texas, USA

    Ioannis Pavlidis

  6. IBM T.J. Watson Research Center, Yorktown Heights, New York, USA

    Rogerio Feris

  7. Purdue University, West Lafayette, Indiana, USA

    Tim McGraw

  8. Side Effects Software, Santa Monica, California, USA

    Mark Elendt

  9. The DiVE, Durham, North Carolina, USA

    Regis Kopper

  10. Texas A&M University, College Station, Texas, USA

    Eric Ragan

  11. Kent State University, Kent, Ohio, USA

    Zhao Ye

  12. Lawrence Berkeley National Laboratory, Berkeley, California, USA

    Gunther Weber

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Vargas, C., Cabezas, I., Branch, J.W. (2015). Stereo Correspondence Evaluation Methods: A Systematic Review. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2015. Lecture Notes in Computer Science(), vol 9475. Springer, Cham. https://doi.org/10.1007/978-3-319-27863-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-27863-6_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-27862-9

  • Online ISBN: 978-3-319-27863-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation