Skip to main content
SpringerLink
Log in

Modeling deviations of rgb-d cameras for accurate depth map and color image registration

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A fundamental step to employ RGB-D cameras is to register color and depth images, whose misalignment may be caused by differences of camera poses and parameters, depth noises, etc. Previous methods mainly devote to more accurate camera calibration, which can only deal with misalignment that are parameterized with camera projection model. Other misalignment, which we call deviations, are more difficult to be measured and modeled. In this paper, we propose a method to model and remove RGB-D camera deviations. First, a specially-designed checkerboard with hollow squares is utilized to measure deviations and camera parameters, it takes advantage of the regularity of corner arrangements and can achieve high accuracy even with noisy depth inputs. Second, we propose a general deviation model to deal with irregular deviations that can not be handled by RGB-D camera projection model. Third, we introduce a registration method that incorporates the estimated deviation model to well register color and depth information. As demonstrated in the experiments, comparing with manufacturer’s calibration and some state-of-the-art algorithms, our approach produces significant better accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Basso F, Pretto A, Menegatti E (2014) Unsupervised intrinsic and extrinsic calibration of a camera-depth sensor couple. In: 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp 6244–6249

  2. Basso F, Menegatti E, Pretto A (2017) Robust intrinsic and extrinsic calibration of rgb-d cameras. arXiv:1701.05748

  3. Bradski G, Kaehler A (2008) Learning openCV: Computer vision with the openCV library. O’Reilly Media, Inc.

  4. Burrus N (2011) Kinect calibration. http://nicolas.burrus.name/index.php/Reasearch/KinectCalibration

  5. Canessa A, Chessa M, Gibaldi A, Sabatini SP, Solari F (2014) Calibrated depth and color cameras for accurate 3d interaction in a stereoscopic augmented reality environment. J Vis Commun Image Represent 25(1):227–237

    Article  Google Scholar 

  6. Castaneda V, Mateus D, Navab N (2011) SLAM Combining tof and high-resolution cameras. In: 2011 IEEE Workshop on Applications of Computer Vision (WACV). IEEE, pp 672–678

  7. Chow JC, Lichti DD (2013) Photogrammetric bundle adjustment with self-calibration of the primesense 3d camera technology: Microsoft kinect. IEEE Access 1:465–474

    Article  Google Scholar 

  8. De Boor C, De Boor C, Mathématicien E-U, De Boor C, De Boor C (1978) A practical guide to splines, vol 27. Springer, New York

    Book  MATH  Google Scholar 

  9. Fleet DJ, Pajdla T, Schiele B, Tuytelaars T (eds) (2014) Computer vision, vol 8693. Springer, Berlin

  10. Frank B, Schmedding R, Stachniss C, Teschner M, Burgard W (2010) Learning deformable object models for mobile robot navigation using depth cameras and a manipulation robot. In: Proceedings of the IEEE International Conference on Robotics & Automation (ICRA)

  11. Fuchs S, Hirzinger G (2008) Extrinsic and depth calibration of tof-cameras. In: CVPR of the 2008 IEEE Conference on Computer Vision and Pattern Recognition 2008. IEEE, pp 1–6

  12. Fuchs H, State A, Bazin J-C (2014) Immersive 3d telepresence. Computer 47(7):46–52

    Article  Google Scholar 

  13. Fusiello A, Trucco E, Verri A (2000) A compact algorithm for rectification of stereo pairs. Mach Vis Appl 12(1):16–22

    Article  Google Scholar 

  14. Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with microsoft kinect sensor: a review. IEEE T Cybernetics 43(5):1318–1334

    Article  Google Scholar 

  15. Harris C, Stephens M (1988) A combined corner and edge detector. In: Alvey Vision Conference, vol 15. Citeseer, p 50

  16. Hartley RI (1992) Estimation of relative camera positions for uncalibrated cameras. In: Computer vision—ECCV’92. Springer, pp 579–587

  17. Henry P, Krainin M, Herbst E, Ren X, Fox D (2010) Rgb-d mapping: using depth cameras for dense 3d modeling of indoor environments. In: the 12th international symposium on experimental robotics (ISER)

  18. Herrera D, Kannala J, Heikkilä J (2011) Accurate and practical calibration of a depth and color camera pair. In: Computer analysis of images and patterns, pp 437–445

  19. Herrera C, Kannala J, Heikkilä J, et al. (2012) Joint depth and color camera calibration with distortion correction. IEEE Trans Pattern Anal Mach Intell 34 (10):2058–2064

    Article  Google Scholar 

  20. Hickson S, Birchfield S, Essa I, Christensen H (2014) Efficient hierarchical graph-based segmentation of rgbd videos. In: 2014 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 344–351

  21. Hong GS, Kim BG, Kwon KK (2014) Efficient depth map estimation method based on gradient weight cost aggregation strategy for distributed video sensor networks. Int J Distrib Sens Netw 10(1):326029

    Article  Google Scholar 

  22. Hong G-S, Koo M-S, Saha A, Kim B-G (2016) Efficient local stereo matching technique using weighted guided image filtering (wgif). In: 2016 IEEE international conference on consumer electronics (ICCE). IEEE, pp 484–485

  23. Hung M-F, Miaou S-G, Chiang C-Y (2013) Dual edge-confined inpainting of 3d depth map using color image’s edges and depth image’s edges. In: Signal and information processing association annual summit and conference (APSIPA), 2013 Asia-Pacific. IEEE, pp 1–9

  24. Jebari I, Filliat D (2012) Color and depth-based superpixels for background and object segmentation. Procedia Eng 41:1307–1315

    Article  Google Scholar 

  25. Jung J, Jeong Y, Park J, Ha H, Kim JD, Kweon I-S (2011) A novel 2.5 d pattern for extrinsic calibration of tof and camera fusion system. In: 2011 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 3290–3296

  26. Jung J, Lee J-Y, Jeong Y, Kweon IS (2015) Time-of-flight sensor calibration for a color and depth camera pair. IEEE Trans Pattern Anal Mach Intell 37(7):1501–1513

    Article  Google Scholar 

  27. Khoshelham K, Elberink SO (2012) Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12(2):1437–1454

    Article  Google Scholar 

  28. Konolige K (2010) Projected texture stereo. In: IEEE international conference on robotics and automation (ICRA), pp 148–155

  29. Kramer J, Burrus N, Echtler F, Daniel HC, Parker M (2012) Hacking the Kinect, vol 268. Springer, Berlin

    Book  Google Scholar 

  30. Kurt Konolige PM Technical description of Kinect calibration. http://wiki.ros.org/kinect_calibration/technical

  31. Lachat E, Macher H, Landes T, Grussenmeyer P (2015) Assessment and calibration of a rgb-d camera (kinect v2 sensor) towards a potential use for close-range 3d modeling. Remote Sens 7(10):13070–13097

    Article  Google Scholar 

  32. Lai K, Bo L, Ren X, Fox D (2011) A large-scale hierarchical multi-view rgb-d object dataset. In: 2011 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp 1817–1824

  33. Mac Aodha O, Campbell ND, Nair A, Brostow GJ (2012) Patch based synthesis for single depth image super-resolution. In: Computer vision–ECCV 2012. Springer, pp 71–84

  34. Matsuo K, Aoki Y (2015) Depth image enhancement using local tangent plane approximations. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  35. Mikhelson IV, Lee PG, Sahakian AV, Wu Y, Katsaggelos AK (2014) Automatic, fast, online calibration between depth and color cameras. J Vis Commun Image Represent 25(1):218–226

    Article  Google Scholar 

  36. Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison AJ, Kohi P, Shotton J, Hodges S, Fitzgibbon A (2011) Kinectfusion: Real-time dense surface mapping and tracking. In: 2011 10th IEEE international symposium on Mixed and augmented reality (ISMAR). IEEE, pp 127–136

  37. Nguyen CV, Izadi S, Lovell D (2012) Modeling kinect sensor noise for improved 3d reconstruction and tracking. In: 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT). IEEE, pp 524–530

  38. Ramey A, González-Pacheco V, Salichs MA (2011) Integration of a low-cost rgb-d sensor in a social robot for gesture recognition. In: Proceedings of the 6th International Conference on Human-Robot Interaction. ACM, pp 229–230

  39. Raposo C, Barreto JP, Nunes U (2013) Fast and accurate calibration of a kinect sensor. In: 2013 International Conference on 3D vision-3DV 2013. IEEE, pp 342–349

  40. Rusu RB, Cousins S (2011) 3D is here: point cloud library (pcl). In: 2011 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp 1–4

  41. Scaramuzza D, Harati A, Siegwart R (2007) Extrinsic self calibration of a camera and a 3d laser range finder from natural scenes. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp 4164–4169

  42. Seitz SM, Curless B, Diebel J, Scharstein D, Szeliski R (2006) A comparison and evaluation of multi-view stereo reconstruction algorithms. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), vol 1. IEEE, pp 519–528

  43. Shao L, Han J, Xu D, Shotton J (2013) Computer vision for RGB-d sensors: Kinect and its applications. IEEE T Cybernetics 43(5):1314–1317

    Article  Google Scholar 

  44. Shapiro L (2002) Stockman g: computer vision. In: Prentice Hall

  45. Shaw JR (2004) Quickfill: an efficient flood fill algorithm, The Code Project

  46. Silberman N, Fergus R (2011) Indoor scene segmentation using a structured light sensor. In: IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp 601–608

  47. Smisek J, Jancosek M, Pajdla T (2013) 3d with kinect. In: Consumer Depth Cameras for Computer Vision, pp 3–25

  48. Song X, Zhong F, Wang Y, Qin X (2014) Estimation of kinect depth confidence through self-training. Vis Comput 30(6-8):855–865

    Article  Google Scholar 

  49. Staranowicz A, Brown GR, Morbidi F, Mariottini GL (2013) Easy-to-use and accurate calibration of rgb-d cameras from spheres. In: Pacific-Rim Symposium on Image and Video Technology, pp 265–278

  50. Staranowicz A N, Brown GR, Morbidi F et al (2015) Practical and accurate calibration of RGB-D cameras using spheres. Comput Vis Image Underst 137:102–114

    Article  Google Scholar 

  51. Unser M, Aldroubi A, Eden M (1993) B-spline signal processing. i. theory. IEEE Trans Signal Process 41(2):821–833

    Article  MATH  Google Scholar 

  52. Vandevenn L (2004) Sample implementations for recursive and non-recursive classic and scanline flood fill

  53. Wang Y, Zhong F, Peng Q, Qin X (2014) Depth map enhancement based on color and depth consistency. Vis Comput 30(10):1157–1168. [Online]. Available: https://doi.org/10.1007/s00371-013-0896-z

    Article  Google Scholar 

  54. Woodford O, Torr P, Reid I, Fitzgibbon A (2009) Global stereo reconstruction under second-order smoothness priors. IEEE Trans Pattern Anal Mach Intell 31(12):2115–2128

    Article  Google Scholar 

  55. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

    Article  Google Scholar 

  56. Zhang Q, Pless R (2004) Extrinsic calibration of a camera and laser range finder (improves camera calibration). In: Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems 2004 (IROS 2004), vol 3. IEEE, pp 2301–2306

  57. Zhang C, Zhang Z (2014) Calibration between depth and color sensors for commodity depth cameras. In: Computer Vision and Machine Learning with RGB-d Sensors, pp 47–64

Download references

Acknowledgments

863 program of China (No.2015AA016405), NSF of China (Nos. 61672326, 61572290), and Shandong Science and Technology Development Plan (No. 2013G0020601).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Shandong Uiversity, Jinan, Shandong, 250101, People’s Republic of China

    Xibin Song, Fan Zhong & Xueying Qin

  2. School of Computer Science and Engineering, Nanyang Technological University, Singapore, 639798, Singapore

    Jianmin Zheng

Authors
  1. Xibin Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianmin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fan Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xueying Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xueying Qin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, X., Zheng, J., Zhong, F. et al. Modeling deviations of rgb-d cameras for accurate depth map and color image registration. Multimed Tools Appl 77, 14951–14977 (2018). https://doi.org/10.1007/s11042-017-5081-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5081-3

Keywords

Search

Navigation