Skip to main content
SpringerLink
Log in

Elaborate Scene Reconstruction with a Consumer Depth Camera

  • Research Article
  • Published:
International Journal of Automation and Computing Aims and scope Submit manuscript

Abstract

A robust approach to elaborately reconstruct the indoor scene with a consumer depth camera is proposed in this paper. In order to ensure the accuracy and completeness of 3D scene model reconstructed from a freely moving camera, this paper proposes new 3D reconstruction methods, as follows: 1) Depth images are processed with a depth adaptive bilateral filter to effectively improve the image quality; 2) A local-to-global registration with the content-based segmentation is performed, which is more reliable and robust to reduce the visual odometry drifts and registration errors; 3) An adaptive weighted volumetric method is used to fuse the registered data into a global model with sufficient geometrical details. Experimental results demonstrate that our approach increases the robustness and accuracy of the geometric models which were reconstructed from a consumer-grade depth camera.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman, A. Davison, A. Fitzgibbon. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, ACM, Santa Barbara, USA, pp. 559–568, 2011.

    Google Scholar 

  2. R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, A. Fitzgibbon. KinectFusion: Real-time dense surface mapping and tracking. In Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality, IEEE, Basel, Switzerland, pp. 127–136, 2011.

    Google Scholar 

  3. B. Curless, M. Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, USA, pp. 303–312, 1996.

    Google Scholar 

  4. S. Rusinkiewicz, M. Levoy. Efficient variants of the ICP algorithm. In Proceedings of the 3rd International Conference on 3D Digital Imaging and Modeling, IEEE, Quebec, Canada, pp. 145–152, 2001.

    Chapter  Google Scholar 

  5. C. Kerl, J. Sturm, D. Cremers. Robust odometry estimation for RGB-D cameras. In Proceedings of IEEE International Conference on Robotics and Automation, IEEE, Karlsruhe, Germany, pp. 3748–3754, 2013.

    Google Scholar 

  6. F. Steinbrücker, J. Sturm, D. Cremers. Realtime visual odometry from dense RGB-D images. In Proceedings of IEEE International Conference on Computer Vision Workshops, IEEE, Barcelona, Spain, pp. 719–722, 2011.

    Google Scholar 

  7. T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. J. Leonard, J. McDonald. Real-time large-scale dense RGBD SLAM with volumetric fusion. International Journal of Robotics Research, vol. 34, no. 4–5, pp. 598–626, 2015.

    Article  Google Scholar 

  8. J. Huai, Y. Zhang, A. Yilmaz. Real-time large scale 3D reconstruction by fusing Kinect and IMU data. In Proceedings of ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, ISPRS, La Grande Motte, France, vol. II-3/W5, pp. 491–496, 2015.

    Google Scholar 

  9. M. Niessner, A. Dai, M. Fisher. Combining inertial navigation and ICP for real-time 3D surface reconstruction. Eurographics, E. Galin, M. Wand, Eds., Strasbourg, France: The Eurographics Association, pp. 13–16, 2014.

    Google Scholar 

  10. K. H. Yang, W. S. Yu, X. Q. Ji. Rotation estimation for mobile robot based on Single-axis gyroscope and monocular camera. International Journal of Automation and Computing, vol. 9, no. 3, pp. 292–298, 2012.

    Article  Google Scholar 

  11. S. Choi, Q. Y. Zhou, V. Koltun. Robust reconstruction of indoor scenes. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Boston, USA, pp. 5556–5565, 2015.

    Google Scholar 

  12. T. Whelan, M. Kaess, J. J. Leonard, J. McDonald. Deformation-based loop closure for large scale dense RGBD SLAM. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Tokyo, Japan, pp. 548–555, 2013.

    Google Scholar 

  13. Q. Y. Zhou, S. Miller, V. Koltun. Elastic fragments for dense scene reconstruction. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Sydney, Australia, pp. 473–480, 2013.

    Google Scholar 

  14. A. Chatterjee, V. M. Govindu. Noise in structured-light stereo depth cameras: Modeling and its applications. arXiv: 1505.01936, 2015.

    Google Scholar 

  15. K. Khoshelham. Accuracy analysis of Kinect depth data. In Proceedings of International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, ISPRS, Calgary, Canada, vol. XXXVIII-5/W12, pp. 133–138, 2011.

    Google Scholar 

  16. K. Khoshelham, S. O. Elberink. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors, vol. 12, no. 2, pp. 1437–1454, 2012.

    Article  Google Scholar 

  17. C. V. Nguyen, S. Izadi, D. Lovell. Modeling Kinect sensor noise for improved 3D reconstruction and tracking. In Proceedings of the 2nd International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission, IEEE, Zurich, Switzerland, pp. 524–530, 2012.

    Google Scholar 

  18. J. Smisek, M. Jancosek, T. Pajdla. 3D with Kinect. In Proceedings of IEEE International Conference on Computer Vision Workshops, IEEE, Barcelona, Spain, pp. 1154–1160, 2011.

    Google Scholar 

  19. C. Tomasi, R. Manduchi. Bilateral filtering for gray and color images. In Proceedings of the 6th International Conference on Computer Vision, IEEE, Bombay, India, pp. 839–846, 1998.

    Google Scholar 

  20. J. X. Xiao, A. Owens, A. Torralba. SUN3D: A database of big spaces reconstructed using SFM and object labels. In Proceedings of IEEE International Conference on Computer Vision, IEEE, Sydney, Australia, pp. 1625–1632, 2013.

    Google Scholar 

  21. Q. F. Chen, V. Koltun. Fast MRF optimization with application to depth reconstruction. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Columbus, USA, pp. 3914–3921, 2014.

    Google Scholar 

  22. J. Kopf, M. F. Cohen, D. Lischinski, M. Uyttendaele. Joint bilateral upsampling. ACM Transactions on Graphics, vol. 26, no. 3, Article number 96, 2007.

    Article  Google Scholar 

  23. T. Whelan, M. Kaess, M. Fallon, H. Johannsson, J. Leonard, J. McDonald. Kintinuous: Spatially extended KinectFusion. In Proceedings of RSS Workshop on RGB-D: Advanced Reasoning with Depth Cameras, Sydney, Australia, pp. 3–14, 2012.

    Google Scholar 

  24. A. Zeng, S. Song, M. Niessner, M. Fisher, J. X. Xiao, T. Funkhouser. 3DMatch: Learning the matching of local 3D geometry in range scans. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Puerto Rico, USA, 2017.

    Google Scholar 

  25. M. Halber, T. Funkhouser. Structured global registration of RGB-D scans in indoor environments. arXiv:1607.08539, 2016.

    Google Scholar 

  26. S. Parker, P. Shirley, Y. Livnat, C. Hansen, P. P. Sloan. Interactive ray tracing for isosurface rendering. In Proceedings of the Conference on Visualization, IEEE, North Carolina, USA, pp. 233–238, 1998.

    Google Scholar 

  27. W. E. Lorensen, H. E. Cline. Marching cubes: A high resolution 3D surface construction algorithm. ACM SIGGRAPH Computer Graphics, vol. 21, no. 4, pp. 163–169, 1987.

    Article  Google Scholar 

  28. M. Zollhöfer, A. Dai, M. Innmann, C. L. Wu, M. Stamminger, C. Theobalt, M. Niessner. Shading-based refinement on volumetric signed distance functions. ACM Transactions on Graphics, vol. 34, no. 4, Article number 96, 2015.

    Article  MATH  Google Scholar 

  29. Q. Y. Zhou, V. Koltun. Dense scene reconstruction with points of interest. ACM Transactions on Graphics, vol. 32, no. 4, Article number 112, 2013.

    MATH  Google Scholar 

  30. Q. Y. Zhou, J. Park, V. Koltun. Fast global registration. In Proceedings of 14th European Conference on Computer Vision, Amsterdam, The Netherlands, pp. 766–782, 2016.

    Google Scholar 

  31. R Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, W. Burgard. g2o: A general framework for graph optimization. In Proceedings of IEEE International Conference on Robotics and Automation, IEEE, Shanghai, China, pp. 3607–3613, 2011.

    Google Scholar 

  32. T. Whelan, S. Leutenegger, R. F. Salas-Moreno, B. Glocker, A. J. Davison. ElasticFusion: Dense SLAM without a pose graph. In Proceedings of Robotics: Science and Systems, Rome, Italy, 2015.

    Google Scholar 

  33. A. Handa, T. Whelan, J. McDonald, A. J. Davison. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In Proceedings of IEEE International Conference on Robotics and Automation, IEEE, Hong Kong, China, pp. 1524–1531, 2014.

    Google Scholar 

  34. C. Kerl, J. Sturm, D. Cremers. Dense visual SLAM for RGB-D cameras. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Tokyo, Japan, pp. 2100–2106, 2013.

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National Key Technologies R & D Program (No. 2016YFB0502002) and in part by National Natural Science Foundation of China (Nos. 61472419, 61421004 and 61572499).

Author information

Authors and Affiliations

  1. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100080, China

    Jian-Wei Li, Wei Gao & Yi-Hong Wu

  2. University of Chinese Academy of Sciences, Beijing, 100049, China

    Jian-Wei Li, Wei Gao & Yi-Hong Wu

Authors
  1. Jian-Wei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi-Hong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Gao.

Additional information

Recommended by Associate Editor Jangmyung Lee

Jian-Wei Li received the B. Sc. degree in measurement, control technology and instrument, the M. Sc. degree in detection technology and automatic equipment from Beijing Jiaotong University, China in 2008 and 2011, respectively. She is currently a Ph.D. degree candidate in pattern recognition and intelligent system from National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China.

Her research interests include 3D reconstruction from images and SLAM technology.

Wei Gao received the B. Sc. degree in computational mathematics, the M. Sc. degree in pattern recognition and intelligent system from Shanxi University and the Ph.D. degree in pattern recognition and intelligent system from Institute of Automation, Chinese Academy of Sciences, China in 2002, 2005 and 2008, respectively. Since July 2008, he has joined Robot Vision Group of National Laboratory of Pattern Recognition, where he is currently an associate professor.

His research interests include 3D reconstruction from images and SLAM technology.

Yi-Hong Wu received the Ph.D. degree in geometric invariants and applications from the Institute of Systems Science, Chinese Academy of Sciences, China in 2001. She is currently a professor at Institute of Automation, Chinese Academy of Sciences, China.

Her research interests include vision geometry, image matching, camera calibration, camera pose determination, SLAM, and their applications.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, JW., Gao, W. & Wu, YH. Elaborate Scene Reconstruction with a Consumer Depth Camera. Int. J. Autom. Comput. 15, 443–453 (2018). https://doi.org/10.1007/s11633-018-1114-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-018-1114-2

Keywords

Search

Navigation