Abstract
Purpose
Calibration and registration are the first steps for augmented reality and mixed reality applications. In the medical field, the calibration between an RGB-D camera and a C-arm fluoroscope is a new topic which introduces challenges.
Method
A convenient and efficient calibration phantom is designed by combining the traditional calibration object of X-ray images with a checkerboard plane. After the localization of the 2D marker points in the X-ray images and the corresponding 3D points from the RGB-D images, we calculate the projection matrix from the RGB-D sensor coordinates to the X-ray, instead of estimating the extrinsic and intrinsic parameters simultaneously.
Validation
In order to evaluate the effect of every step of our calibration process, we performed five experiments by combining different steps leading to the calibration. We also compared our calibration method to Tsai’s method to evaluate the advancement of our solution. At last, we simulated the process of estimating the rotation movement of the RGB-D camera using MATLAB and demonstrate that calculating the projection matrix can reduce the angle error of the rotation.
Results
A RMS reprojection error of 0.5 mm is achieved using our calibration method which is promising for surgical applications. Our calibration method is more accurate when compared to Tsai’s method. Lastly, the simulation result shows that using a projection matrix has a lower error than using intrinsic and extrinsic parameters in the rotation estimation.
Conclusions
We designed and evaluated a 3D/2D calibration method for the combination of a RGB-D camera and a C-arm fluoroscope.
Similar content being viewed by others
References
Kersten-Oertel M, Jannin P, Collins DL (2012) DVV: a taxonomy for mixed reality visualization in image guided surgery. IEEE Trans Vis Comput Graph 18(2):332–352
Navab N, Blum T, Wang L, Okur A, Wendler T (2012) First deployments of augmented reality in operating rooms. Comput IEEE Comput Soc 7:48–55
Ewers R, Schicho K, Wagner A, Undt G, Seemann R, Figl M, Truppe M (2005) Seven years of clinical experience with teleconsultation in craniomaxillofacial surgery. J Oral Maxillofac Surg 63:1447–1454
Johnson LG, Edwards P, Hawkes D (2003) Surface transparency makes stereo overlays unpredictable: the implications for augmented reality. Stud Health Technol Inform 94:131–136
Bauer S, Seitel A, Hofmann H, Blum T, Wasza J, Balda M, Meinzer HP, Navab N, Hornegger J, Maier-Hein L (2013) Real-time range imaging in health care: a survey. In: Grzegorzek M, Theobalt C, Koch R, Kolb A (eds) Time-of-flight and depth imaging. Sensors, algorithms, and applications. Springer, Berlin, Heidelberg, pp 228–254
Wang XL, Stolka PJ, Boctor E, Hager G, Choti M (2012) The Kinect as an interventional tracking system. In: David R. Holmes III, Kenneth H. Wong (eds) SPIE medical imaging 2012: image-guided procedures, robotic interventions and modeling, vol 8316. SPIE, San Diego, California, USA, pp 83160U-83160U-6
Kontschieder P, Dorn JF, Morrison C, Corish R, Zikic D, Sellen A, D’Souza M, Kamm CP, Burggraaff J, Tewarie P, Vogel T, Azzarito M, Glocker B, Dahlke P, Polman C, Kappos L, Uitdehaag B, Criminisi A (2014) Quantifying progression of multiple sclerosis via classification of depth videos. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention—MICCAI 2014, vol 8674. Springer International Publishing, Switzerland, pp 429–437
Alnowami M, Alnwaimi B, Tahavori F, Copland M, Wells K (2012) A quantitative assessment of using the Kinect for Xbox360 for respiratory surface motion tracking. In: David R. Holmes III, Kenneth H. Wong (eds) SPIE medical imaging, international society for optics and photonics, SPIE, San Diego, California, USA, pp 83161T–83161T-10
Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334
Gábor B (2015) Camera calibration with opencv.http://www.swarthmore.edu/NatSci/mzucker1/opencv-2.4.10-docs/doc/tutorials/calib3d/camera_calibration/camera_calibration.html#cameracalibrationopencv. Accessed 24 Nov 2015
Yaniv Z, Joskowicz L, Simkin A, Garza-Jinich M, Milgrom C (1998) Fluoroscopic image processing for computer-aided orthopaedic surgery. In: Wells WM, Colchester A, Delp S (eds) Medical image computing and computer-assisted interventation—MICCAI’98, vol 1496. Springer, Berlin, Heidelberg, pp 325–334
Livyatan H, Yaniv Z, Joskowicz L (2002) Robust automatic C-arm calibration for fluoroscopy-based navigation: a practical approach. In: Dohi T, Kikinis R (eds) Medical image computing and computer-assisted intervention–MICCAI 2002, vol 2489. Springer, Berlin, Heidelberg, pp 60–68
Meng C, Zhang J, Zhou F, Wang T (2014) New method for geometric calibration and distortion correction of conventional C-arm. Comput Biol Med 52:49–56
Shechter G, Devernay F, Coste-Manière E, McVeigh ER (2002) Temporal tracking of 3D coronary arteries in projection angiograms. In: Medical imaging 2002, international society for optics and photonics. pp 612–623
Shechter G, Shechter B, Resar JR, Beyar R (2005) Prospective motion correction of X-ray images for coronary interventions. IEEE Trans Med Imaging 24(4):441–450
Tsai R (1987) A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lens. IEEE J Robot Autom 3:323–344
Jung J, Lee JY, Jeong Y, Kweon IS (2014) Time-of-flight sensor calibration for a color and depth camera pair. IEEE Trans Pattern Anal Mach Intell 37(7):1501–1513
Shime H, Adelsberger R, Kim JD, Rhee SM, Rhee T, Sim JY, Gross M, Kim C (2012) Time-of-flight sensor and color camera calibration for multi-view acquisition. Vis Comput 28:1139–1151
Zhang C, Zhang Z (2014) Calibration between depth and color sensors for commodity depth cameras. In: Shao L, Han J, Kohli P, Zhang Z (eds) Computer vision and machine learning with RGB-D sensors. Springer International Publishing, Switzerland, pp 47–64
Herrera C, Kannala J, Heikkilä J (2012) Joint depth and color camera calibration with distortion correction. IEEE Trans Pattern Anal Mach Intell 34(10):2058–8828
Raposo C, Barreto JP, Nunes U (2013) Fast and accurate calibration of a Kinect sensor. In: International conference on 3D vision—3DV. pp 342–349
Horaud R, Dornaika F (1995) Hand-eye calibration. Int J Robot Res 14(3):195–210
Mitschke MM, Navab N (2000) Recovering projection geometry: how a cheap camera can outperform an expensive stereo system. In: IEEE conference on computer vision and pattern recognition (CVPR), vol. 1. pp 193–200
Navab N, Bani-Hashemi A, Nadar MS, Wiesent K, Durlak P, Brunner T, Barth K, Graumann R (1998) 3D Reconstruction from projection matrices in a C-arm based 3D-angiography system. In: Wells WM, Colchester A, Delp S (eds) Medical image computing and computer assisted interventation—MICCAI’98, vol 1496. Springer, Berlin, Heidelberg, pp 119–129
Yang L, Zhang L, Dong H, Alelaiwi A, Saddik AE (2015) Evaluating and improving the depth accuracy of Kinect for Windows v2. IEEE Sens J 15(8):4275–4285
Kedgley AE, Fox AV, Jenkyn TR (2012) Image intensifier distortion correction for fluoroscopic RSA: the need for independent accuracy assessment. J Appl Clin Med Phys 13(1):197–204
Navab N, Mitschke M (2001) Method and apparatus using a virtual detector for three-dimensional reconstruction from X-ray images. Patent US 6236704; Filing date: June 30, 1999; Issue date: May 22
Huhle B, Schairer T, Jenke P, Strasser W (2008) Robust non-local denosing of colored depth data. In: IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW’08). pp 1–7
Essmaeel K, Gallo L, Damiani E, De Pietro G, Dipanda A (2012) Temporal denoising of Kinect depth data. In: 2012 Eighth international conference on signal image technology and internet based systems (SITIS). pp 47–52
Hartley R, Zisserman A (2004) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, Cambridge
Moré JJ (1978) The Levenberg–Marquardt algorithm: implementation and theory. In: Watson GA (ed) Numerical analysis. Springer, Berlin, Heidelberg, pp 105–116
Horn BKP (2000) Tsai’s camera calibration method revisited. http://people.csail.mit.edu/bkph/articles/Tsai_Revisited.pdf
Golub GH, Van Loan CF (1996) Matrix computation, 3rd edn. Johns Hopkins University Press, Baltimore
Horn BKP (2008) Finding the nearest orthonormal matrix. http://people.csail.mit.edu/bkph/articles/Nearest_Orthonormal_Matrix.pdf
Huynh DQ (2009) Metrics for 3D rotation: comparison and analysis. J Math Imaging Vis 35:155–164
Pauly O, Diotte B, Fallavollita P, Weidert S, Euler E, Navab N (2015) Machine learning-based augmented reality for improved surgical scene understanding. Comput Med Imaging Graph 41:55–60
Pauly O, Diotte B, Habert S, Weidert S, Euler E, Fallavollita P, Navab N (2014). Relevance-based visualization to improve surgeon perception. In: Stoyanov D, Collins DL, Sakuma I, Abolmaesumi P, Jannin P (eds) Information processing in computer-assisted interventions, vol 8498. Springer International Publishing, Switzerland, pp 178–185
Pauly O, Katouzian A, Eslami A, Fallavollita P, Navab N (2012). Supervised classification for customized intraoperative augmented reality visualization. In: 2012 IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, pp 311–312
Wieczorek M, Aichert A, Fallavollita P, Kutter O, Ahmadi A, Wang L, Navab N (2011) Interactive 3D visualization of a single-view X-ray image. In: Fichtinger G, Martel A, Peters T (eds) Medical image computing and computer-assisted intervention-MICCAI 2011, vol 6891. Springer, Berlin, Heidelberg, pp 73–80
Erat O, Pauly O, Weidert S, Thaller P, Euler E, Mutschler W, et al (2013) How a surgeon becomes superman by visualization of intelligently fused multi-modalities. In: David R. Holmes, Ziv R. Yaniv (eds) SPIE medical imaging. International Society for Optics and Photonics, vol 8671. SPIE, Florida, USA, pp 86710L–86710L-6
Wang L, Fallavollita P, Brand A, Erat O, Weidert S, Thaller PH et al (2012) Intra-op measurement of the mechanical axis deviation: an evaluation study on 19 human cadaver legs. In: Ayache N, Delingette H, Golland P, Mori K (eds) Medical image computing and computer-assisted intervention—MICCAI 2012. Springer, Berlin, Heidelberg, pp 609–616
Diotte B, Fallavollita P, Wang L, Weidert S, Euler E, Thaller P, Navab N (2015) Multi-modal intra-operative navigation during distal locking of intramedullary nails. Med Imaging IEEE Trans 34(2):487–495
Londei R, Esposito M, Diotte B, Weidert S, Euler E, Thaller P, et al (2014) The ‘augmented’circles: a video-guided solution for the down-the-beam positioning of IM nail holes. In: Stoyanov D, Collins DL, Sakuma I, Abolmaesumi P, Jannin P (eds) Information processing in computer-assisted interventions, vol 8498. Springer International Publishing, Switzerland, pp 100–107
Londei R, Esposito M, Diotte B, Weidert S, Euler E, Thaller P, et al (2015) Intra-operative augmented reality in distal locking. Int J Comput Assist Radiol Surg 10(9):1395–1403
Acknowledgments
This work was partly supported by the China Scholarship Council.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Human and animal rights
This article does not contain any studies with human participants or animals performed by any of the authors.
Informed consent
Statement of informed consent was not applicable since the manuscript does not contain any patient data.
Rights and permissions
About this article
Cite this article
Wang, X., Habert, S., Ma, M. et al. Precise 3D/2D calibration between a RGB-D sensor and a C-arm fluoroscope. Int J CARS 11, 1385–1395 (2016). https://doi.org/10.1007/s11548-015-1347-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-015-1347-2