skip to main content
10.1145/2435264.2435278acmconferencesArticle/Chapter ViewAbstractPublication PagesfpgaConference Proceedingsconference-collections
research-article

Video-rate stereo matching using markov random field TRW-S inference on a hybrid CPU+FPGA computing platform

Published: 11 February 2013 Publication History

Abstract

We demonstrate a video-rate stereo matching system implemented on a hybrid CPU+FPGA platform (Convey HC-1). Emerging applications such as 3D gesture recognition and automotive navigation demand fast and high quality stereo vision. We describe a custom hardware-accelerated Markov Random Field inference system for this task. Starting from a core architecture for streaming tree-reweighted message passing (TRW-S) inference, we describe the end-to-end system engineering needed to move from this single frame message update to full stereo video. We partition the stereo matching procedure across the CPU and the FPGAs, and apply both function-level pipelining and frame-level parallelism to achieve the required speed. Experimental results show that our system achieves a speed of 12 frames per second for challenging video stereo matching tasks. We note that this appears to be the first implementation of TRW-S inference at video rates, and that our system is also significantly faster than several recent GPU implementations of similar stereo inference methods based on belief propagation (BP).

References

[1]
Choi, J. and Rutenbar, R. A. 2012. Hardware implementation of MRF MAP inference on an FPGA platform. In Proc. IEEE 22nd Int. Conf. Field Programmable Logic and Applications. 209--216.
[2]
Ko, D. --I. and Agarwal, G. 2011. Gesture recognition-first step toward 3D UIs? EETimes. http://www.eetimes.com/design/embedded/4231057/Gesture-recognition-first-step-toward-3D-UIs-
[3]
Gong, H., Sim, J., Likhachev, M. and Shi, J. 2011. Multi-hypothesis motion planning for visual object tracking. In Proc. IEEE Int. Conf. Computer Vision. 619--626.
[4]
Koller, D. and Friedman, N. 2009. Probabilistic graphical model. MIT press.
[5]
Boykov, Y. and Veksler, O. and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI).11:1222--1239.
[6]
Pearl, J. 1988. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann
[7]
Tappen, M. F. and Freeman, W. T. 2003. Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters. In Proc. IEEE 9th Int. Conf. Computer Vision. 900--907.
[8]
Sun, J., Zheng, N. N., and Shum, H. Y. 2003. Stereo matching using belief propagation. IEEE Trans. PAMI. 7:787--800.
[9]
Li, Y. and Huttenlocher, D.P. 2008. Learning for stereo vision using the structured support vector machine. In Proc. IEEE Int. Conf. Computer Vision and Pattern Recognition (CVPR). 1--8.
[10]
Szeliski, R., et al. 2008. A comparative study of energy minimization methods for Markov random fields with smoothness based priors. IEEE Trans. PAMI. 6:1068--1080.
[11]
Felzenszwalb, P. F. and Huttenlocher, D. P. 2004. Efficient belief propagation for early vision. In Proc. IEEE Int. Conf. CVPR. 1261--1268.
[12]
Gonzalez, J., Low, Y. and Guestrin, C. 2009. Residual splash for optimally parallelizing belief propagation. In Proc. 12th Int. Conf. Artificial Intelligence and Statistics.
[13]
Kanade, T., Kano, H., Kimura, S., Yoshida, A. and Oda, K. 1995. Development of a video-rate stereo machine. In Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems. 95--100.
[14]
Park, S., Chen, C., and Jeong, H. 2007. VLSI architecture for MRF based stereo matching. In Proc. 7th Int. Conf. Embedded Computer Systems: Architectures, Modeling, and Simulation. 55--64.
[15]
Park, J., Lee, S., and Yoo, H.J. 2010. A 30fps stereo matching processor based on belief propagation with disparity-parallel PE array architecture. In Proc. IEEE Int. Symp. Circuits and Systems. 453--456.
[16]
Liang, C. --K., Cheng, C. C., Lai, Y. C., Chen, L. G., and Chen, H. H. 2011. Hardware-efficient belief propagation. IEEE Trans. Circuits and Systems for Video Technology. 5:525--537
[17]
Yang, Q., Wang, L., Yang, R., Wang, S., Liao, M., and Nister, D. 2006. Real-time global stereo matching using hierarchical belief propagation. The British Machine Vision Conference. 989--998.
[18]
Xiang, X., Zhang, M., Li, G., He, Y., and Pan, Z. 2012. Real-time stereo matching based on fast belief propagation. Machine Vision and Applications, 1--9.
[19]
Wainwright, M., Jaakkola, T. S., and Willsky, A. S. 2005. Map estimation via agreement on trees: message-passing and linear programming. IEEE Trans. Information Theory. 11:3697--3717.
[20]
Kolmogorov, V. 2006. Convergent tree-reweighted message passing for energy minimization. IEEE Trans. PAMI. 10:568--1583.
[21]
Birchfield, S. and Tomasi, C. 1998. A pixel dissimilarity measure that is insensitive to image sampling. IEEE Trans. PAMI. 4:401--406.
[22]
Hirschmuller H. and Scharstein, D. 2009 Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. on PAMI. 9:1582--1599.
[23]
Convey computer. 2012. Convey Reference Manual. http://www.conveycomputer.com.
[24]
Bakos, J.D. 2010. High-performance heterogeneous computing with the convey hc-1. Computing in Science & Engineering. 12(6):80--87.
[25]
zlib: a massively spiffy yet delicately unobtrusive compression library. http://www.zlib.net.
[26]
libpng. http://www.libpng.org.
[27]
Lam, M. 1988. Software pipelining: an effective scheduling technique for VLIW machines. In Proc. ACM SIGPLAN Conf. Programming Language design and Implementation. 7:318--328.
[28]
Convey computer. 2012. Convey Programmers Guide. http://www.conveycomputer.com.
[29]
D. Scharstein and R. Szeliski. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1/2/3):7--42.
[30]
Stereo movie sample. http://www.stereomaker.net/sample/index.html.

Cited By

View all
  • (2023)Unconventional computing based on magnetic tunnel junctionApplied Physics A10.1007/s00339-022-06365-4129:4Online publication date: 3-Mar-2023
  • (2021)Brain-Inspired Hardware Solutions for Inference in Bayesian NetworksFrontiers in Neuroscience10.3389/fnins.2021.72808615Online publication date: 2-Dec-2021
  • (2019)Multi-Modal ISAR Object Recognition using Adaptive Deep Relation Learning2019 International Conference on Wireless Communications Signal Processing and Networking (WiSPNET)10.1109/WiSPNET45539.2019.9032812(48-53)Online publication date: Mar-2019
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
FPGA '13: Proceedings of the ACM/SIGDA international symposium on Field programmable gate arrays
February 2013
294 pages
ISBN:9781450318877
DOI:10.1145/2435264
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 February 2013

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. convey hybrid-core computing platform
  2. markov random field energy minimization methods
  3. tree-reweighted message passing
  4. video-rate stereo matching

Qualifiers

  • Research-article

Conference

FPGA '13
Sponsor:

Acceptance Rates

Overall Acceptance Rate 125 of 627 submissions, 20%

Upcoming Conference

FPGA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)3
  • Downloads (Last 6 weeks)0
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Unconventional computing based on magnetic tunnel junctionApplied Physics A10.1007/s00339-022-06365-4129:4Online publication date: 3-Mar-2023
  • (2021)Brain-Inspired Hardware Solutions for Inference in Bayesian NetworksFrontiers in Neuroscience10.3389/fnins.2021.72808615Online publication date: 2-Dec-2021
  • (2019)Multi-Modal ISAR Object Recognition using Adaptive Deep Relation Learning2019 International Conference on Wireless Communications Signal Processing and Networking (WiSPNET)10.1109/WiSPNET45539.2019.9032812(48-53)Online publication date: Mar-2019
  • (2016)Error Resilient and Energy Efficient MRF Message-Passing-Based Stereo MatchingIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2015.243733124:3(897-908)Online publication date: 1-Mar-2016
  • (2016)Video-Rate Stereo Matching Using Markov Random Field TRW-S Inference on a Hybrid CPU+FPGA Computing PlatformIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2015.239719826:2(385-398)Online publication date: 1-Feb-2016
  • (2016)A real-time global stereo-matching on FPGAMicroprocessors & Microsystems10.1016/j.micpro.2016.08.00547:PB(419-428)Online publication date: 1-Nov-2016
  • (2015)Fast hierarchical implementation of sequential tree-reweighted belief propagation for probabilistic inference2015 25th International Conference on Field Programmable Logic and Applications (FPL)10.1109/FPL.2015.7293934(1-8)Online publication date: Sep-2015
  • (2014)A robust message passing based stereo matching kernel via system-level error resiliency2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2014.6855226(8331-8335)Online publication date: May-2014
  • (2014)Energy-efficient accelerator architecture for stereo image matching using approximate computing and statistical error compensation2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP)10.1109/GlobalSIP.2014.7032077(55-59)Online publication date: Dec-2014
  • (2013)FPGA acceleration of Markov Random Field TRW-S inference for stereo matchingProceedings of the Eleventh ACM/IEEE International Conference on Formal Methods and Models for Codesign10.5555/3041405.3041495(139-142)Online publication date: 1-Oct-2013

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media