Skip to main content
Springer Nature Link
Log in

Realistic surface geometry reconstruction using a hand-held RGB-D camera

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

In this paper, we have proposed a novel approach for the reconstruction of real object/scene with realistic surface geometry using a hand-held, low-cost, RGB-D camera. To achieve accurate reconstruction, the most important issues to consider are the quality of the geometry information provided and the global alignment method between frames. In our approach, new surface geometry refinement is used to recover finer scale surface geometry from depth data by utilizing high-quality RGB images. In addition, a weighted multi-scale iterative closest point method is exploited to align each scan to the global model accurately. We show the effectiveness of the proposed surface geometry refinement method by comparing it with other depth refinement methods. We also show both the qualitative and quantitative results of reconstructed models by comparing it with other reconstruction methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://www.ar-tracking.com/home/.

  2. http://www.pointclouds.org/news/kinectfusion-open-source.html.

  3. http://www.faro.com/scenect/scenect.

  4. http://www.faro.com/en-us/products/3d-surveying/faro-focus3d/overview.

References

  1. Beeler, T., Bickel, B., Beardsley, P., Sumner, B., Gross, M.: High-quality single-shot capture of facial geometry. ACM Trans. Graph. 29(4), 40:1–40:9 (2010). doi:10.1145/1778765.1778777. http://doi.acm.org

  2. Besl, P., McKay, H.: A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992). doi:10.1109/34.121791

    Article  Google Scholar 

  3. Chen, Y., Medioni, G.: Object modelling by registration of multiple range images. Image Vis. Comput. 10(3), 145–155 (1992). doi:10.1016/0262-8856(92)90066-C

    Article  Google Scholar 

  4. Cui, Y., Schuon, S., Chan, D., Thrun, S., Theobalt, C.: 3d shape scanning with a time-of-flight camera. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), pp 1173 –1180 (2010). doi:10.1109/CVPR.2010.5540082

  5. Ferstl, D., Reinbacher, C., Ranftl, R., Ruether, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp 993–1000 (2013). doi:10.1109/ICCV.2013.127

  6. Freedman SAMMAY Barak (2008) Depth mapping using projected patterns. http://www.freepatentsonline.com/y2008/0240502.html

  7. Garcia, F., Mirbach, B., Ottersten, B., Grandidier, F., Cuesta, A.: Pixel weighted average strategy for depth sensor data fusion. In: 2010 17th IEEE International Conference on Image Processing (ICIP), pp 2805–2808 (2010). doi:10.1109/ICIP.2010.5651112

  8. Garcia, F., Aouada, D., Mirbach, B., Solignac, T., Ottersten, B.: A new multi-lateral filter for real-time depth enhancement. In: 2011 8th IEEE International Conference on Advanced Video and Signal-Based Surveillance (AVSS), pp 42 –47 (2011). doi:10.1109/AVSS.2011.6027291

  9. Geiger, A., Ziegler, J., Stiller, C.: Stereoscan: dense 3d reconstruction in real-time. In: Intelligent Vehicles Symposium. IEEE, New York, pp 963–968 (2011)

  10. Godin, G., Rioux, M., Baribeau, R.: Three-dimensional registration using range and intensity information. Proc. SPIE (1994). doi:10.1117/12.189139

  11. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013). doi:10.1109/TPAMI.2012.213

    Article  Google Scholar 

  12. Herrera, C.D., Kannala, J., Heikkil, J.: Joint depth and color camera calibration with distortion correction. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 10, pp. 2058–2064 (2012). doi:10.1109/TPAMI.2012.125

  13. Hirschmuller, H., Scharstein, D.: Evaluation of cost functions for stereo matching. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR ’07, pp 1–8 (2007). doi:10.1109/CVPR.2007.383248

  14. Kazhdan, M., Hoppe, H.: Screened poisson surface reconstruction. ACM Trans. Graph. 32(3), 29:1–29:13 (2013). doi:10.1145/2487228.2487237

  15. Khoshelham, K., Elberink, S.O.: Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12(2), 1437–1454 (2012). doi:10.3390/s120201437

  16. Khoshelham, K., Dos Santos, D., Vosselman, G.: Generation and weighting of 3d point correspondences for improved registration of rgb-d data. In: ISPRS Annals of the Photogrammetry and Remote Sensing and Spatial Information Sciences, vol. II-5/W2 (2013)

  17. Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. In: ACM SIGGRAPH 2007 Papers. ACM, New York (2007). SIGGRAPH ’07. doi:10.1145/1275808.1276497

  18. Lee, K.R., Nguyen, T.: Robust tracking and mapping with a handheld RGB-D camera. In: 2014 IEEE Winter Conference on Applications of Computer Vision (WACV), pp 1120–1127 (2014). doi:10.1109/WACV.2014.6835732

  19. Lee, K.R., Khoshabeh, R., Nguyen, T.: Sampling-based robust multi-lateral filter for depth enhancement. In: 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), pp 1124–1128 (2012)

  20. Low, K.L.: Linear least-squares optimization for point-to-plane ICP surface registration. Tech. Rep. TR04-004. Department of Computer Science, University of North Carolina at Chapel Hill (2004)

  21. Lu, J., Min, D., Pahwa, R., Do, M.: A revisit to mrf-based depth map super-resolution and enhancement. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 985–988 (2011). doi:10.1109/ICASSP.2011.5946571

  22. Mohr, R., Quan, L., Veillon, F.: Relative 3d reconstruction using multiple uncalibrated images, pp. 543–548 (1995)

  23. Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.: Kinectfusion: real-time dense surface mapping and tracking. In: Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality. IEEE Computer Society, Washington, DC, ISMAR ’11, pp 127–136 (2011). doi:10.1109/ISMAR.2011.6092378

  24. Nguyen, C., Izadi, S., Lovell, D.: Modeling kinect sensor noise for improved 3d reconstruction and tracking. In: 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), pp 524–530 (2012). doi:10.1109/3DIMPVT.2012.84

  25. Oggier, T.: An all-solid-state optical range camera for 3d real-time imaging with sub-centimeter depth resolution (swissranger). Proc. SPIE 5249(65), 534–545 (2004)

    Article  Google Scholar 

  26. Park, J., Kim, H., Tai, Y.W., Brown, M., Kweon, I.: High quality depth map upsampling for 3d-tof cameras. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp 1623–1630 (2011). doi:10.1109/ICCV.2011.6126423

  27. Pollefeys, M.: Self calibration and metric 3d reconstruction from uncalibrated image sequences. PhD thesis, Leuven (1999)

  28. Rusinkiewicz, S., Levoy, M.: Efficient variants of the icp algorithm. In: International Conference on 3-D Digital Imaging and Modeling (2001)

  29. Rusinkiewicz, S., Hall-Holt, O., Levoy, M.: Real-time 3d model acquisition. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, SIGGRAPH ’02, pp. 438–446 (2002). doi:10.1145/566570.566600

  30. Seitz, S., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 519–528 (2006). doi:10.1109/CVPR.2006.19

  31. Slabaugh, G., Culbertson, B., Malzbender, T., Schafer, R.: A Survey of Methods for Volumetric Scene Reconstruction from Photographs. pp. 81–100 (2001). http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.17.4793

  32. Tenedorio, D., Fecho, M., Schwartzhaupt, J., Pardridge, R., Lue, J., Schulze, J.P.: Capturing geometry in real-time using a tracked microsoft kinect, pp. 82,890A–82,890A–14 (2012). doi:10.1117/12.912211

  33. Tong, J., Zhou, J., Liu, L., Pan, Z., Yan, H.: Scanning 3d full human bodies using kinects. In: IEEE Transactions on Visualization and Computer Graphics 18, 643–650 (2012). doi:10.1109/TVCG.2012.56

  34. Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR ’07, pp 1–8 (2007). doi:10.1109/CVPR.2007.383211

Download references

Acknowledgments

This work is supported in part by NSF Grant CCF-1160832.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, 92037, USA

    Kyoung-Rok Lee & Truong Nguyen

Authors
  1. Kyoung-Rok Lee
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Truong Nguyen
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Kyoung-Rok Lee.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, KR., Nguyen, T. Realistic surface geometry reconstruction using a hand-held RGB-D camera. Machine Vision and Applications 27, 377–385 (2016). https://doi.org/10.1007/s00138-016-0747-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-016-0747-9

Keywords