Skip to main content
SpringerLink
Log in

Real-time extraction of surface patches with associated uncertainties by means of Kinect cameras

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In this paper, we present our work on GPU-based real-time extraction of surface patches by means of Kinect cameras. This paper makes four contributions: (1) we derive an uncertainty model for pixel-wise depth reconstruction on Kinect cameras; (2) we implement a real-time algorithm for surface patch (here called ‘texlet’) extraction based on Kinect depth data on a GPU. For that we compare and evaluate different implementation alternatives. (3) Based on (1) we derive and implement an appropriate uncertainty model for texlets which is also computed in real-time. (4) We investigate and quantify the effect of interferences on the depth extraction process when using multiple Kinect cameras. By these contributions we present insights into the processing of depth and how to achieve higher precision reconstructions by means of Kinect cameras as well as extend their use for higher level visual processing. The introduced algorithms are available in the C++ vision library CoViS.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. See http://www.xbox.com/kinect.

  2. See e.g., the LibFreenect driver http://openkinect.org.

  3. Point Cloud Library, http://pointclouds.org.

  4. Robot Operating System, http://www.ros.org.

  5. See http://www.openni.org/.

  6. see e.g., CULA tools (http://culatools.com).

  7. Some of the solutions might be coinciding or zero.

  8. Some points in a neighborhood might be discarded as a result of outlier rejection.

  9. This is similar to the Unscented Transform method [10].

  10. If only one or two Kinects were used, it would not be possible to get all relations, due to an incomplete scene representation.

  11. See http://www.covis.org/.

References

  1. Başeski, E., Pugeault, N., Kalkan, S., Bodenhagen, L., Piater, J.H., Krüger, N.: Using multi-modal 3D contours and their relations for vision and robotics. J Vis Commun Image Represent 21(8), 850–864 (2010)

    Article  Google Scholar 

  2. Berger, K., Ruhl, K., Brümmer C., Schröder, Y., Scholz, A., Magnor, M.: Markerless motion capture using multiple color-depth sensors. In: Proceedings of Vision, Modeling and Visualization (VMV) 2011, pp. 317–324, October 2011

  3. Carceroni, R.L., Kutalakos, K.N.: Multi-view scene capture by surfel sampling: From video streams to non-rigid 3D motion, shape and reflectance. In: Proceedings of Eighth IEEE International Conference on Computer Vision, ICCV, vol. 2, pp. 60–67 (2001)

  4. Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Google Scholar 

  5. Gottfried, J.-M., Fehr, J., Garbe, C.: Computing range flow from multi-modal Kinect data. In: Advances in Visual Computing, vol. 6938 of Lecture Notes in Computer Science, pp. 758–767. Springer, Berlin (2011)

  6. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)

  7. Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: RGB-D mapping: Using depth cameras for dense 3D modeling of indoor environments. In: RGB-D: Advanced Reasoning with Depth Cameras Workshop in conjunction with RSS (2010)

  8. Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., Stuetzle, W.: Surface reconstruction from unorganized points. In: Proceedings of ACM SIGGRAPH (1992)

  9. Jessen, J.B., Pilz, F., Kraft, D., Pugeault, N., Krüger, N.: Accumulation of different visual feature descriptors in a coherent framework. In: Scandinavian Conference on Image Analysis (SCIA) (2011)

  10. Julier, S.J., Idak Industries: The scaled unscented transformation. In: Proceedings of IEEE Amer. Control Conf, pp. 4555–4559 (2002)

  11. Khoshelham, K.: Accuracy analysis of Kinect depth data. In: ISPRS Workshop Laser Scanning, vol. XXXVIII (2011)

  12. Kim, Y.M., Chan, D., Theobalt, C., Thrun, S.: Design and calibration of a multi-view TOF sensor fusion system. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008. CVPRW’08, pp. 1–7 (2008)

  13. Kim, Y.M., Chan, D., Theobalt, C., Thrun, S.: Design and calibration of a multi-view TOF sensor fusion system. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008. CVPRW ’08 (2008)

  14. Kitagawa, G.: Monte Carlo filter and smoother for non-gaussian nonlinear state space models. J. Comput. Graph. Stat. 5(1), 1–25 (1996)

    Google Scholar 

  15. Klasing, K., Althoff, D., Wollherr, D., Buss, M.: Comparison of surface normal estimation methods for range sensing applications. In: Proceedings of the 2009 IEEE International Conference on Robotics and Automation, ICRA’09, pp. 1977–1982, IEEE Press, Piscataway, NJ, USA (2009)

  16. Kramer, J., Burrus, N., Herrera D.C., Echtler, F., Parker, M.: Hacking the Kinect. Apress (2012)

  17. Middleton, L., Sivaswamy, J.: Hexagonal Image Processing : A Practical Approach. Springer, London (2005)

  18. Murray, D., Little, J.J.: Patchlets: Representing stereo vision data with surface elements. In: Seventh IEEE Workshops on Application of Computer Vision, 2005 (WACV/MOTIONS ’05 ), vol. 1, pp. 192–199 (2005)

  19. Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.: Kinectfusion: Real-time dense surface mapping and tracking. In: ISMAR ’11 Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pp. 127–136 (2011)

  20. Pfister, H., Zwicker, M., van Baar, J., Gross, M.: Surfels: Surface elements as rendering primitives. In: ACM SIGGRAPH, pp. 335–342 (2000)

  21. Popović, M., Kootstra, G., Jørgensen, J.A., Kragic, D., Krüger, N.: Grasping unknown objects using an early cognitive vision system for general scene understanding. In: Proceedings of the International Conference on Intelligent Robots and Systems (IROS), September 25–30, San Francisco, CA (2011)

  22. Pugeault, N., Krüger, N.: Temporal accumulation of oriented visual features. J. Vis. Commun. Image Represent. 22(2), 153–163 (2011)

    Article  Google Scholar 

  23. Quigley, M., Gerkey, B., Conley, K., Faust, J., Foote, T., Leibs, J., Berger, E., Wheeler, R., Ng, A.: ROS: An open-source Robot Operating System. In: ICRA Workshop on Open Source Software

  24. Reynolds, M., Dobos, J., Peel, L., Weyrich, T., Brostow, G.J.: Capturing time-of-flight data with confidence. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2011)

  25. Rusu, R.B., Cousins, S.: 3D is here: Point Cloud Library (PCL). In: IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, May 9–13 2011

  26. Saenko, K., Karayev, S., Jia, Y., Shyr, A., Janoch, A., Long, J., Fritz, M., Darrell, T.: Practical 3-D object detection using category and instance-level appearance models. In: IROS (2011)

  27. Schuon, S., Theobalt, C., Davis, J., Thrun, S.: LidarBoost: depth superresolution for ToF 3D shape scanning. In: Proceedings of IEEE CVPR (2009)

  28. Smisek, J., Jancosek, M., Pajdla, T.: 3D with Kinect. In: IEEE Workshop on Consumer Depth Cameras for Computer Vision, November 2011

  29. Xia, L., Chen, C.-C, Aggarwal, J.K.: Human detection using depth information by Kinect. In: International Workshop on Human Activity Understanding from 3D Data in conjunction with CVPR (HAU3D), June 2011

Download references

Acknowledgments

This work has been supported by the IntellAct project (FP7-ICT-269959).

Author information

Authors and Affiliations

  1. The Mærsk Mc-Kinney Møller Institute, University of Southern Denmark, Odense, Denmark

    Søren Maagaard Olesen, Simon Lyder, Dirk Kraft, Norbert Krüger & Jeppe Barsøe Jessen

Authors
  1. Søren Maagaard Olesen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simon Lyder
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dirk Kraft
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Norbert Krüger
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jeppe Barsøe Jessen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jeppe Barsøe Jessen.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Olesen, S.M., Lyder, S., Kraft, D. et al. Real-time extraction of surface patches with associated uncertainties by means of Kinect cameras. J Real-Time Image Proc 10, 105–118 (2015). https://doi.org/10.1007/s11554-012-0261-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-012-0261-x

Keyword

Search

Navigation