skip to main content
10.1145/2789116.2789148acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicdscConference Proceedingsconference-collections
research-article

A passive RGBD sensor for accurate and real-time depth sensing self-contained into an FPGA

Published: 08 September 2015 Publication History

Abstract

In this paper we describe the strategy adopted to design, from scratch, an embedded RGBD sensor for accurate and dense depth perception on a low-cost FPGA. This device infers, at more than 30 Hz, dense depth maps according to a state-of-the-art stereo vision processing pipeline entirely mapped into the FPGA without buffering partial results on external memories. The strategy outlined in this paper enables accurate depth computation with a low latency and a simple hardware design. On the other hand, it poses major constraints to the computing structure of the algorithms that fit with this simplified architecture and thus, in this paper, we discuss the solutions devised to overcome these issues. We report experimental results concerned with practical application scenarios in which the proposed RGBD sensor provides accurate and real-time depth sensing suited for the embedded vision domain.

References

[1]
C. Banz, H. Blume, and P. Pirsch. Real-time semi-global matching disparity estimation on the gpu. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, pages 514--521, Nov 2011.
[2]
C. Banz, S. Hesselbarth, H. Flatt, H. Blume, and P. Pirsch. Real-time stereo vision system using semi-global matching disparity estimation: Architecture and fpga-implementation. In ICSAMOS, pages 93--101, 2010.
[3]
L. di Stefano, M. Marchionni, and S. Mattoccia. A fast area-based stereo matching algorithm. Image Vision Comput., 22(12):983--1005, 2004.
[4]
S. Gehrig and C. Rabe. Real-time semi-global matching on the cpu. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, pages 85--92, June 2010.
[5]
S. K. Gehrig, F. Eberli, and T. Meyer. A real-time low-power stereo vision engine using semi-global matching. In ICVS, pages 134--143, 2009.
[6]
A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
[7]
K. T. Gribbon, C. T. Johnston, and D. G. Bailey. A real-time fpga implementation of a barrel distortion correction algorithm with bilinear interpolation. In In Proceedings of the Image and Vision Computing New Zealand, pages 408--413, 2003.
[8]
H. Hirschmüller. Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell., 30(2):328--341, 2008.
[9]
H. Hirschmüller and D. Scharstein. Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell., 31(9):1582--1599, 2009.
[10]
D. Honegger, H. Oleynikova, and M. Pollefeys. Real-time and low latency embedded computer vision hardware based on a combination of fpga and mobile cpu. In Intelligent Robots and Systems (IROS 2014), 2014 IEEE/RSJ International Conference on, pages 4930--4935, Sept 2014.
[11]
S. Mattoccia. Stereo vision algorithms for fpgas. In The 9th IEEE Embedded Vision Workshop, CVPR 2013 Workshop, 2013.
[12]
D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. Journal Computer Vision, 47(1/2/3):7--42, 2002.
[13]
K. Schmid and H. Hirschmuller. Stereo vision and imu based real-time ego-motion and depth image computation on a handheld device. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 4671--4678, May 2013.
[14]
F. Schumacher and T. Greiner. Extension and fpga architecture of the generalized hough transform for real-time stereo correspondence. In Design and Architectures for Signal and Image Processing (DASIP), 2013 Conference on, pages 223--229, Oct 2013.
[15]
Y. Shan, Y. Hao, W. Wang, Y. Wang, X. Chen, H. Yang, and W. Luk. Hardware acceleration for an accurate stereo vision system using mini-census adaptive support region. ACM Trans. Embed. Comput. Syst., 13(4s):132:1--132:24, apr 2014.
[16]
F. Stein. Efficient Computation of Optical Flow Using the Census Transform. In Proceedings of the 26th DAGM Symposium, pages 79--86, 2004.
[17]
B. Tippetts, D. Lee, K. Lillywhite, and J. Archibald. Review of stereo vision algorithms and their suitability for resource-limited systems. Journal of Real-Time Image Processing, 2013.
[18]
C. Ttofis and T. Theocharides. Towards accurate hardware stereo correspondence: A real-time fpga implementation of a segmentation-based adaptive support weight algorithm. In Design, Automation Test in Europe Conference Exhibition (DATE), 2012, pages 703--708, March 2012.
[19]
R. Zabih and J. Woodfill. Non-parametric local transforms for computing visual correspondence. In Proceedings of the Third European Conference-Volume II on Computer Vision - Volume II, ECCV '94, pages 151--158, 1994.

Cited By

View all
  • (2024)Fruit modeling and application based on 3D imaging technology: a reviewJournal of Food Measurement and Characterization10.1007/s11694-024-02480-318:6(4120-4136)Online publication date: 12-Mar-2024
  • (2023)Cell-Based Refinement Processor Utilizing Disparity Characteristics of Road Environment for SGM-Based Stereo Vision SystemsIEEE Access10.1109/ACCESS.2023.333864911(138122-138140)Online publication date: 2023
  • (2021)Robotic Computing on FPGAsSynthesis Lectures on Computer Architecture10.2200/S01101ED1V01Y202105CAC05616:1(1-218)Online publication date: 29-Jun-2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICDSC '15: Proceedings of the 9th International Conference on Distributed Smart Cameras
September 2015
225 pages
ISBN:9781450336819
DOI:10.1145/2789116
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

  • Escuela Técnica superier de Ingeniería Informática, Universidad de Seville, Spain: Escuela Técnica superier de Ingeniería Informática, Universidad de Seville, Spain

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 September 2015

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. FPGA
  2. RGBD
  3. embedded
  4. real-time
  5. stereo vision

Qualifiers

  • Research-article

Conference

ICDSC '15
Sponsor:
  • Escuela Técnica superier de Ingeniería Informática, Universidad de Seville, Spain

Acceptance Rates

ICDSC '15 Paper Acceptance Rate 43 of 48 submissions, 90%;
Overall Acceptance Rate 92 of 117 submissions, 79%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)10
  • Downloads (Last 6 weeks)1
Reflects downloads up to 28 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Fruit modeling and application based on 3D imaging technology: a reviewJournal of Food Measurement and Characterization10.1007/s11694-024-02480-318:6(4120-4136)Online publication date: 12-Mar-2024
  • (2023)Cell-Based Refinement Processor Utilizing Disparity Characteristics of Road Environment for SGM-Based Stereo Vision SystemsIEEE Access10.1109/ACCESS.2023.333864911(138122-138140)Online publication date: 2023
  • (2021)Robotic Computing on FPGAsSynthesis Lectures on Computer Architecture10.2200/S01101ED1V01Y202105CAC05616:1(1-218)Online publication date: 29-Jun-2021
  • (2021)Continual Adaptation for Deep StereoIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.3075815(1-1)Online publication date: 2021
  • (2021)A Survey of FPGA-Based Robotic ComputingIEEE Circuits and Systems Magazine10.1109/MCAS.2021.307160921:2(48-74)Online publication date: Oct-2022
  • (2021)Real‐time multi‐window stereo matching algorithm with fuzzy logicIET Computer Vision10.1049/cvi2.1203115:3(208-223)Online publication date: 23-Mar-2021
  • (2020)Linking Points With Labels in 3D: A Review of Point Cloud Semantic SegmentationIEEE Geoscience and Remote Sensing Magazine10.1109/MGRS.2019.29376308:4(38-59)Online publication date: Dec-2020
  • (2020)Self-adapting Confidence Estimation for StereoComputer Vision – ECCV 202010.1007/978-3-030-58586-0_42(715-733)Online publication date: 30-Nov-2020
  • (2019)Real-Time Highly Accurate Dense Depth on a Power Budget Using an FPGA-CPU Hybrid SoCIEEE Transactions on Circuits and Systems II: Express Briefs10.1109/TCSII.2019.290916966:5(773-777)Online publication date: May-2019
  • (2019)Leveraging Confident Points for Accurate Depth Refinement on Embedded Systems2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW.2019.00025(158-167)Online publication date: Jun-2019
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media