research-article

A passive RGBD sensor for accurate and real-time depth sensing self-contained into an FPGA

Authors:

Stefano Mattoccia,

Matteo PoggiAuthors Info & Claims

ICDSC '15: Proceedings of the 9th International Conference on Distributed Smart Cameras

Pages 146 - 151

https://doi.org/10.1145/2789116.2789148

Published: 08 September 2015 Publication History

Abstract

In this paper we describe the strategy adopted to design, from scratch, an embedded RGBD sensor for accurate and dense depth perception on a low-cost FPGA. This device infers, at more than 30 Hz, dense depth maps according to a state-of-the-art stereo vision processing pipeline entirely mapped into the FPGA without buffering partial results on external memories. The strategy outlined in this paper enables accurate depth computation with a low latency and a simple hardware design. On the other hand, it poses major constraints to the computing structure of the algorithms that fit with this simplified architecture and thus, in this paper, we discuss the solutions devised to overcome these issues. We report experimental results concerned with practical application scenarios in which the proposed RGBD sensor provides accurate and real-time depth sensing suited for the embedded vision domain.

References

[1]

C. Banz, H. Blume, and P. Pirsch. Real-time semi-global matching disparity estimation on the gpu. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pages 514--521, Nov 2011.

[2]

C. Banz, S. Hesselbarth, H. Flatt, H. Blume, and P. Pirsch. Real-time stereo vision system using semi-global matching disparity estimation: Architecture and fpga-implementation. In ICSAMOS, pages 93--101, 2010.

[3]

L. di Stefano, M. Marchionni, and S. Mattoccia. A fast area-based stereo matching algorithm. Image Vision Comput., 22(12):983--1005, 2004.

[4]

S. Gehrig and C. Rabe. Real-time semi-global matching on the cpu. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, pages 85--92, June 2010.

[5]

S. K. Gehrig, F. Eberli, and T. Meyer. A real-time low-power stereo vision engine using semi-global matching. In ICVS, pages 134--143, 2009.

Digital Library

[6]

A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.

Digital Library

[7]

K. T. Gribbon, C. T. Johnston, and D. G. Bailey. A real-time fpga implementation of a barrel distortion correction algorithm with bilinear interpolation. In In Proceedings of the Image and Vision Computing New Zealand, pages 408--413, 2003.

[8]

H. Hirschmüller. Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell., 30(2):328--341, 2008.

Digital Library

[9]

H. Hirschmüller and D. Scharstein. Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell., 31(9):1582--1599, 2009.

Digital Library

[10]

D. Honegger, H. Oleynikova, and M. Pollefeys. Real-time and low latency embedded computer vision hardware based on a combination of fpga and mobile cpu. In Intelligent Robots and Systems (IROS 2014), 2014 IEEE/RSJ International Conference on, pages 4930--4935, Sept 2014.

[11]

S. Mattoccia. Stereo vision algorithms for fpgas. In The 9th IEEE Embedded Vision Workshop, CVPR 2013 Workshop, 2013.

Digital Library

[12]

D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. Journal Computer Vision, 47(1/2/3):7--42, 2002.

Digital Library

[13]

K. Schmid and H. Hirschmuller. Stereo vision and imu based real-time ego-motion and depth image computation on a handheld device. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 4671--4678, May 2013.

[14]

F. Schumacher and T. Greiner. Extension and fpga architecture of the generalized hough transform for real-time stereo correspondence. In Design and Architectures for Signal and Image Processing (DASIP), 2013 Conference on, pages 223--229, Oct 2013.

[15]

Y. Shan, Y. Hao, W. Wang, Y. Wang, X. Chen, H. Yang, and W. Luk. Hardware acceleration for an accurate stereo vision system using mini-census adaptive support region. ACM Trans. Embed. Comput. Syst., 13(4s):132:1--132:24, apr 2014.

Digital Library

[16]

F. Stein. Efficient Computation of Optical Flow Using the Census Transform. In Proceedings of the 26th DAGM Symposium, pages 79--86, 2004.

[17]

B. Tippetts, D. Lee, K. Lillywhite, and J. Archibald. Review of stereo vision algorithms and their suitability for resource-limited systems. Journal of Real-Time Image Processing, 2013.

[18]

C. Ttofis and T. Theocharides. Towards accurate hardware stereo correspondence: A real-time fpga implementation of a segmentation-based adaptive support weight algorithm. In Design, Automation Test in Europe Conference Exhibition (DATE), 2012, pages 703--708, March 2012.

Digital Library

[19]

R. Zabih and J. Woodfill. Non-parametric local transforms for computing visual correspondence. In Proceedings of the Third European Conference-Volume II on Computer Vision - Volume II, ECCV '94, pages 151--158, 1994.

Digital Library

Cited By

Wang JXie ZMao PSun MGuo J(2024)Fruit modeling and application based on 3D imaging technology: a reviewJournal of Food Measurement and Characterization10.1007/s11694-024-02480-318:6(4120-4136)Online publication date: 12-Mar-2024
https://doi.org/10.1007/s11694-024-02480-3
Choi COh HHan JShin J(2023)Cell-Based Refinement Processor Utilizing Disparity Characteristics of Road Environment for SGM-Based Stereo Vision SystemsIEEE Access10.1109/ACCESS.2023.333864911(138122-138140)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3338649
Liu SWan ZYu BWang Y(2021)Robotic Computing on FPGAsSynthesis Lectures on Computer Architecture10.2200/S01101ED1V01Y202105CAC05616:1(1-218)Online publication date: 29-Jun-2021
https://doi.org/10.2200/S01101ED1V01Y202105CAC056
Show More Cited By

Index Terms

A passive RGBD sensor for accurate and real-time depth sensing self-contained into an FPGA

Recommendations

Accurate hardware-based stereo vision
Research highlights
► Real-Time Stereo Vision using FPGAs. ► Gradient-based Census Transform combined with SAD allows for high accuracy. ► Sparse computation of the Hamming Distance reduces computational complexity. ► Highly ...
Abstract
To enable both accurate and fast real-time stereo vision in embedded systems, we propose a novel stereo matching algorithm that is designed for high efficiency when realized in hardware. We evaluate its accuracy using the Middlebury ...
FPGA based real-time on-road stereo vision system

Accurate and Real-Time stereo vision is an essential need for many computer vision applications, such as On-Road stereo vision system. In this paper, a fast and accurate system for On-Road stereo vision application is presented. In order to achieve this ...
Low-cost FPGA stereo vision system for real time disparity maps calculation

Several applications demand efficient hardware implementations of stereo vision systems in order to furnish real time three-dimensional measurements. This paper proposes a complete fast low-cost stereo vision system that performs stereo image ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICDSC '15: Proceedings of the 9th International Conference on Distributed Smart Cameras

September 2015

225 pages

ISBN:9781450336819

DOI:10.1145/2789116

General Chairs:
Ricardo Carmona-Galán,
Ángel Rodríguez-Vázquez
IMSE-CNM (CSIC-Universidad de Sevilla), Spain

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Escuela Técnica superier de Ingeniería Informática, Universidad de Seville, Spain: Escuela Técnica superier de Ingeniería Informática, Universidad de Seville, Spain

In-Cooperation

SIGBED: ACM Special Interest Group on Embedded Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 September 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICDSC '15

Sponsor:

Escuela Técnica superier de Ingeniería Informática, Universidad de Seville, Spain

ICDSC '15: International Conference on distributed Smart Cameras

September 8 - 11, 2015

Seville, Spain

Acceptance Rates

ICDSC '15 Paper Acceptance Rate 43 of 48 submissions, 90%;

Overall Acceptance Rate 92 of 117 submissions, 79%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
186
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang JXie ZMao PSun MGuo J(2024)Fruit modeling and application based on 3D imaging technology: a reviewJournal of Food Measurement and Characterization10.1007/s11694-024-02480-318:6(4120-4136)Online publication date: 12-Mar-2024
https://doi.org/10.1007/s11694-024-02480-3
Choi COh HHan JShin J(2023)Cell-Based Refinement Processor Utilizing Disparity Characteristics of Road Environment for SGM-Based Stereo Vision SystemsIEEE Access10.1109/ACCESS.2023.333864911(138122-138140)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3338649
Liu SWan ZYu BWang Y(2021)Robotic Computing on FPGAsSynthesis Lectures on Computer Architecture10.2200/S01101ED1V01Y202105CAC05616:1(1-218)Online publication date: 29-Jun-2021
https://doi.org/10.2200/S01101ED1V01Y202105CAC056
Poggi MTonioni ATosi FMattoccia SDi Stefano L(2021)Continual Adaptation for Deep StereoIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.3075815(1-1)Online publication date: 2021
https://doi.org/10.1109/TPAMI.2021.3075815
Wan ZYu BLi TTang JZhu YWang YRaychowdhury ALiu S(2021)A Survey of FPGA-Based Robotic ComputingIEEE Circuits and Systems Magazine10.1109/MCAS.2021.307160921:2(48-74)Online publication date: Oct-2022
https://doi.org/10.1109/MCAS.2021.3071609
Vázquez‐Delgado HPérez‐Patricio MAguilar‐González AArias‐Estrada MPalacios‐Ramos MCamas‐Anzueto JPérez‐Cruz AVelázquez‐Trujillo S(2021)Real‐time multi‐window stereo matching algorithm with fuzzy logicIET Computer Vision10.1049/cvi2.1203115:3(208-223)Online publication date: 23-Mar-2021
https://doi.org/10.1049/cvi2.12031
Xie YTian JZhu X(2020)Linking Points With Labels in 3D: A Review of Point Cloud Semantic SegmentationIEEE Geoscience and Remote Sensing Magazine10.1109/MGRS.2019.29376308:4(38-59)Online publication date: Dec-2020
https://doi.org/10.1109/MGRS.2019.2937630
Poggi MAleotti FTosi FZaccaroni GMattoccia S(2020)Self-adapting Confidence Estimation for StereoComputer Vision – ECCV 202010.1007/978-3-030-58586-0_42(715-733)Online publication date: 30-Nov-2020
https://doi.org/10.1007/978-3-030-58586-0_42
Rahnama OCavallari TGolodetz STonioni AJoy TDi Stefano LWalker STorr P(2019)Real-Time Highly Accurate Dense Depth on a Power Budget Using an FPGA-CPU Hybrid SoCIEEE Transactions on Circuits and Systems II: Express Briefs10.1109/TCSII.2019.290916966:5(773-777)Online publication date: May-2019
https://doi.org/10.1109/TCSII.2019.2909169
Tosi FPoggi MMattoccia S(2019)Leveraging Confident Points for Accurate Depth Refinement on Embedded Systems2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW.2019.00025(158-167)Online publication date: Jun-2019
https://doi.org/10.1109/CVPRW.2019.00025
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten