Skip to main content
Springer Nature Link
Log in

The VLSI implementation of a high-resolution depth-sensing SoC based on active structured light

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

This paper presents the full VLSI implementation of a new high-resolution depth-sensing system on a chip (SoC) based on active infrared structured light, which estimates the 3D scene depth by matching randomized speckle patterns, akin to the Microsoft Kinect. We present a module to enhance the consistency of speckle patterns for robust matching, and hence improve the range of depth estimation. We present a simple and efficient hardware structure for a block-matching-based disparity estimation algorithm, which facilitates rapid generation of disparity maps in real time. For depth estimation from disparity maps, we propose a hardware friendly solution by producing a lookup table from a curve fitted to calculate and calibrate the depth value in a single step, which does not need to explicitly calibrate the parameters of the imaging sensors, such as the length of the baseline. We have implemented these ideas in an end-to-end SoC using FPGA. Compared with the Kinect, our depth-sensing SoC has wider effective ranging limit (0.6–4.5 m), and has an uppermost \(1280\times 1024\) processing capacity with 60 Hz frame frequency. Its depth resolution is 1 mm@0.82 m. Our system is superior to Kinect in terms of operating range, processing frame rate, and resolution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

References

  1. Kinect for Xbox 360. http://www.xbox.com/en-SG/Xbox360/Accessories/Kinect (2015). Accessed Apr 2015

  2. Han, J., Shao, L., Xu, D., et al.: Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans. Cybern. 43(5), 1318–1334 (2013)

    Article  Google Scholar 

  3. Zhang, Z.: Microsoft kinect sensor and its effect. MultiMedia IEEE. 19(2), 4–10 (2012)

    Article  Google Scholar 

  4. Shotton, J., Sharp, T., Kipman, A., et al.: Real-time human pose recognition in parts from single depth images. Commun. ACM 56(1), 116–124 (2013)

    Article  Google Scholar 

  5. Henry, P., Krainin, M., Herbst, E., et al.: RGB-D mapping: using Kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robotics Res. 31(5), 647–663 (2012)

    Article  Google Scholar 

  6. Izadi, S., Kim, D., Hilliges, O., et al.: Kinect Fusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th annual ACM symposium on User interface software and technology. pp. 559–568 (2011)

  7. Ganapathi, V., Plagemann, C., Koller, D., et al.: Real time motion capture using a single time-of-flight camera. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 755–762 (2010)

  8. Bevilacqua, A., Di Stefano, L., Azzari, P.: People tracking using a time-of-flight depth sensor. In: 2006 IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS’06), vol. 6, pp. 89–93 (2006)

  9. Goesele, M., Curless, B., Seitz, S.M.: Multi-view stereo revisited. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 2402–2409. IEEE (2006)

  10. Murray, D., Jennings, C.: Stereo vision based mapping and navigation for mobile robots. In: 1997 IEEE International Conference on Robotics and Automation, vol. 2, pp. 1694–1699. IEEE (1997)

  11. Jain, R., Kasturi, R., Schunck, B.G.: Machine vision, vol. 5. McGraw-Hill, New York (1995)

    Google Scholar 

  12. Geiger, D., Ladendorf, B., Yuille, A.: Occlusions and binocular stereo. Int. J. Comput. Vision 14(3), 211–226 (1995)

    Article  Google Scholar 

  13. Strecha, C., Fransens, R., Van Gool, L.: Combined depth and outlier estimation in multi-view stereo. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 2394–2401. IEEE (2006)

  14. Bradley, D., Boubekeur, T., Heidrich, W.: Accurate multi-view reconstruction using robust binocular stereo and surface meshing. In: 2008 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8. IEEE (2008)

  15. Kanade, T., Yoshida, A., Oda, K., et al.: A stereo machine for video-rate dense depth mapping and its new applications. In: 1996 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR96, pp. 196–202. IEEE (1996)

  16. Gokturk, S.B., Yalcin, H., Bamji, C.: A time-of-flight depth sensor-system description, issues and solutions. In: Conference on Computer Vision and Pattern Recognition Workshop (CVPRW04), pp. 35–35. IEEE (2004)

  17. Foix, S., Alenya, G., Torras, C.: Lock-in time-of-flight (TOF) cameras: a survey. Sensors J. IEEE 11(9), 1917–1926 (2011)

    Article  Google Scholar 

  18. Kim, W., Yibing, W., Ovsiannikov, I., et al.: A 1.5 Mpixel RGBZ CMOS image sensor for simultaneous color and range image capture. In: Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 2012 IEEE International, pp. 392–394. IEEE (2012)

  19. Blais, F.: Review of 20 years of range sensor development. J. Electron. Imaging 13(1), 231–240 (2004)

    Article  Google Scholar 

  20. Huang, P.S., Zhang, S.: 3-d optical measurement using phase shifting based methods. In: Optics East 2005, vol. 6000, no. 1, pp. 2–12. International Society for Optics and Photonics (2005)

  21. DApuzzo, N.: Overview of 3d surface digitization technologies in europe. In: Electronic Imaging 2006, vol. 6056, no. 1, pp. 1–13. International Society for Optics and Photonics (2006)

  22. Gomercic, M., Winter, D.: Projector for an arrangement for three-dimensional optical measurement of objects (2009). US Patent 7,532,332

  23. Geng, J.: Structured-light 3d surface imaging: a tutorial. Adv. Optics Photonics 3(2), 128–160 (2011)

    Article  Google Scholar 

  24. Prime Sense. http://www.primesense.com/ (2011). Accessed October 2012

  25. Garcia, J., Zalevsky, Z.: Range mapping using speckle decorrelation (2008). US Patent 7,433,024

  26. Shpunt, A.: Optical designs for zero order reduction (2008). US Patent App. 12/330,766

  27. Kim, S.Y., Cho, J.H., Koschan, A., et al.: Spatial and temporal enhancement of depth images captured by a time-of-flight depth sensor. In: 2010 20th International Conference on Pattern Recognition (ICPR), pp. 2358–2361. IEEE (2010)

  28. Lai, P., Tian, D., Lopez, P.: Depth map processing with iterative joint multilateral filtering. In: 2010 Picture Coding Symposium (PCS), pp. 9–12. IEEE (2010)

  29. Camplani, M., Salgado, L.: Efficient spatio-temporal hole filling strategy for kinect depth maps. Proc. SPIE 8920, 1–10 (2012)

    Google Scholar 

  30. Maini, R., Aggarwal, H.: A comprehensive review of image enhancement techniques. arXiv preprint arXiv:1003.4053 (2010)

  31. Jourlin, M., Pinoli, J.C.: A model for logarithmic image processing. J. Microsc. 149(1), 21–35 (1988)

    Article  Google Scholar 

  32. Deng, G., Cahill, L., Tobin, G.: The study of logarithmic image processing model and its application to image enhancement. IEEE Trans. Image Process. 4(4), 506–512 (1994)

    Article  Google Scholar 

  33. Abed, K.H., Siferd, R.E.: Cmos vlsi implementation of a lowpower logarithmic converter. IEEE Trans. Comput. 52(11), 1421–1433 (2003)

    Article  Google Scholar 

  34. Imaging, Aptina: MT9M001: 1/2-Inch Megapixel CMOS Digital Image Sensor. MT9M001 DS Rev, vol. 1, 1–27(2004)

  35. Khoshelham, K., Elberink, S.O.: Accuracy and resolution of kinect depth data for indoor mapping applications. Mol. Divers. Preserv. Int. 12(2), 1437–1454 (2012)

    Google Scholar 

Download references

Acknowledgments

This work was supported by “the Fundamental Research Funds for the Central Universities”, NSFC under Grant No. 61228303 and Shaanxi Industrial Projects.

Author information

Authors and Affiliations

  1. Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi, China

    Huimin Yao, Chenyang Ge & Nanning Zheng

  2. Department of Computer Science, Stevens Institute of Technology, Hoboken, NJ, 07030, USA

    Gang Hua

Authors
  1. Huimin Yao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Chenyang Ge
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Gang Hua
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Nanning Zheng
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Chenyang Ge.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yao, H., Ge, C., Hua, G. et al. The VLSI implementation of a high-resolution depth-sensing SoC based on active structured light. Machine Vision and Applications 26, 533–548 (2015). https://doi.org/10.1007/s00138-015-0680-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-015-0680-3

Keywords