Abstract
Purpose
During autopsy, forensic pathologists today mostly rely on visible indication, tactile perception and experience to determine the cause of death. Although computed tomography (CT) data is often available for the bodies under examination, these data are rarely used due to the lack of radiological workstations in the pathological suite. The data may prevent the forensic pathologist from damaging evidence by allowing him to associate, for example, external wounds to internal injuries. To facilitate this, we propose a new multimodal approach for intuitive visualization of forensic data and evaluate its feasibility.
Methods
A range camera is mounted on a tablet computer and positioned in a way such that the camera simultaneously captures depth and color information of the body. A server estimates the camera pose based on surface registration of CT and depth data to allow for augmented reality visualization of the internal anatomy directly on the tablet. Additionally, projection of color information onto the CT surface is implemented.
Results
We validated the system in a postmortem pilot study using fiducials attached to the skin for quantification of a mean target registration error of \(4.4 \pm 1.3\) mm.
Conclusions
The system is mobile, markerless, intuitive and real-time capable with sufficient accuracy. It can support the forensic pathologist during autopsy with augmented reality and textured surfaces. Furthermore, the system enables multimodal documentation for presentation in court. Despite its preliminary prototype status, it has high potential due to its low price and simplicity.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig1_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig2_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig3_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig4_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig5_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig6_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig7_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig8_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig9_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig10_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig11_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-014-1106-9/MediaObjects/11548_2014_1106_Fig12_HTML.jpg)
Similar content being viewed by others
Notes
E.g., in the institute for Forensic Medicine and Traffic Medicine in Heidelberg, almost for every case a CT scan is acquired before autopsy.
We provide design and 3D printing plan publicly at http://www.thingiverse.com/thing:343425.
Bronchoscopy Model Sick Boy http://www.gtsimulators.com.
References
Ampanozi G, Zimmermann D, Hatch GM, Ruder TD, Ross S, Flach PM, Thali MJ, Ebert LC (2012) Format preferences of district attorneys for post-mortem medical imaging reports: understandability, cost effectiveness, and suitability for the courtroom: A questionnaire based study. Legal Med 14(3):116–120
Batcher KE (1968) Sorting networks and their applications. In: Proceedings of the April 30-May 2, Spring Joint Computer Conference, pp 307–314
Bauer S, Seitel A, Hofmann H, Blum T, Wasza J, Balda M, Meinzer HP, Navab N, Hornegger J, Maier-Hein L (2013) Real-time range imaging in health care: a survey. In: Time-of-flight and depth imaging. Sensors, algorithms, and applications. Springer, Berlin, pp 228–254
Besl PJ, Neil DM (1992) A method for registration of 3-d shapes. IEEE Trans Pattern Anal Mach Intell 14(2):239–256
Bhatia A, Patel S, Pantol G, Wu YY, Plitnikas M, Hancock C (2013) Intra and inter-observer reliability of mobile tablet PACS viewer system versus standard PACS viewing station-diagnosis of acute central nervous system events. Open J Radiol 03(02):91–98
Blum T, Kleeberger V, Bichlmeier C, Navab N (2012) mirracle: an augmented reality magic mirror system for anatomy education. In: Virtual reality short papers and posters (VRW), IEEE, pp 115–116
Chetverikov D, Svirko D, Stepanov D, Krsek P (2002) The trimmed iterative closest point algorithm. In: Proceedings of the 16th international conference on pattern recognition, vol 3. IEEE, pp 545–548
Dabrowski JR, Munson EV (2001) Is 100 milliseconds too fast? In: CHI EA ’01 CHI ’01 Extended abstracts on human factors in computing systems. ACM Press, pp 317–318
Danilchenko A, Fitzpatrick JM (2011) General approach to first-order error prediction in rigid point registration. IEEE Trans Med Imaging 30(3):679–693
dos Santos TR, Seitel A, Kilgus T, Suwelack S, Wekerle AL, Kenngott H, Speidel S, Schlemmer HP, Meinzer HP, Heimann T, Maier-Hein L (2014) Pose-independent surface matching for intra-operative soft-tissue marker-less registration. Med Image Anal (accepted subject to minor revisions)
Ebert L, Hatch G, Thali M, Ross S (2013) Invisible touch–control of a DICOM viewer with finger gestures using the kinect depth camera. J Forensic Radiol Imaging 1(1):10–14
Ebert LC, Hatch G, Ampanozi G, Thali MJ, Ross S (2012) You can’t touch this: touch-free navigation through radiological images. Surg Innov 19(3):301–307
Ebert LC, Ptacek W, Breitbeck R, Fürst M, Kronreif G, Martinez RM, Thali M, Flach PM (2014) Virtobot 2.0: the future of automated surface documentation and CT-guided needle placement in forensic medicine. Forensic Sci Med Pathol 10(2):179–186
Ebert LC, Ptacek W, Naether S, Fürst M, Ross S, Buck U, Weber S, Thali M (2009) Virtobot: a multi-functional robotic system for 3D surface scanning and automatic post mortem biopsy. Int J Med Robot Comput Assist Surg 6:18–27
Glocker B, Izadi S, Shotton J, Criminisi A (2013) Real-time RGB-D camera relocalization. In: IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, pp 173–179
Handa A, Newcombe RA, Angeli A, Davison AJ (2012) Real-time camera tracking: when is high frame-rate best? In: Computer vision-ECCV 2012. Springer, pp 222–235
Hartley R, Zisserman A (2003) Multiple view geometry in computer vision. Cambridge University Press, Cambridge
Horn BKP (1987) Closed-form solution of absolute orientation using unit quaternions. J Opt Soc Am A 4(4):629–642
Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, Shotton J, Hodges S, Freeman D, Davison A, Fritzgibbon A (2011) KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th annual ACM symposium on user interface software and technology. ACM, pp 559–568
John S, Poh ACC, Lim TCC, Chan EHY, Chong LR (2012) The iPad tablet computer for mobile on-call radiology diagnosis? Auditing discrepancy in CT and MRI reporting. J Digit Imaging 25(5):628–634
Khoshelham K, Elberink SO (2012) Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12(12):1437–1454
Kilgus T, Franz A, Seitel A, März K, Bartha L, Fangerau M, Mersmann S, Groch A, Meinzer HP, Maier-Hein L (2012) Registration of partially overlapping surfaces for range image based augmented reality on mobile devices. In: SPIE medical imaging. International Society for Optics and Photonics, pp 83160T–83160T
Kilgus T, dos Santos TR, Seitel A, Yung K, Franz AM, Groch A, Wolf I, Meinzer HP, Maier-Hein L (2011) Generation of triangle meshes from time-of-flight data for surface registration. In: Bildverarbeitung für die Medizin 2011. Springer, pp 189–193
Kirk DB, Wen-mei WH (2012) Programming massively parallel processors: a hands-on approach. Newnes
Kramer J, Burrus N, Echtler F, Daniel HC, Parker M (2012) Hacking the Kinect. Springer, New York
Li H, Sumner RW, Pauly M (2008) Global correspondence optimization for non-rigid registration of depth scans. In: Eurographics symposium on geometry processing, vol 27. Wiley, pp 1421–1430
Lorensen WE, Cline HE (1987) Marching cubes: a high resolution 3d surface construction algorithm. In: ACM siggraph computer graphics, vol 21. ACM, pp 163–169
Maier-Hein L, Fangerau M, Meinzer HP, Seitel A (2013) Visualization of anatomical data by augmented reality. US Patent App. 13/891,310
Maier-Hein L, Franz AM, Fangerau M, Schmidt M, Seitel A, Mersmann S, Kilgus T, Groch A, Yung K, dos Santos TR, Meinzer HP (2011) Towards mobile augmented reality for on-patient visualization of medical images. In: Bildverarbeitung für die Medizin 2011, Informatik Aktuell. Springer, pp 389–393
Maier-Hein L, Franz AM, Santos TRD, Schmidt M, Fangerau M, Meinzer HP, Fitzpatrick JM (2012) Convergent iterative closest-point algorithm to accommodate anisotropic and inhomogenous localization error. IEEE Trans Pattern Anal Mach Intell 34(8):1520–1532
Mersmann S, Seitel A, Erz M, Jähne B, Nickel F, Mieth M, Mehrabi A, Maier-Hein L (2013) Calibration of time-of-flight cameras for accurate intraoperative surface reconstruction. Med Phys 40(8):082701
Müller M, Rassweiler MC, Klein J, Seitel A, Gondan M, Baumhauer M, Teber D, Rassweiler JJ, Meinzer HP, Maier-Hein L (2013) Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int J Comput Assist Radiol Surg 8(4):663–675
Myronenko A, Song X (2010) Point set registration: coherent point drift. IEEE Trans Pattern Anal Mach Intell 32(12):2262–2275
Navab N, Traub J, Sielhorst T, Feuerstein M, Bichlmeier C (2007) Action- and workflow-driven augmented reality for computer-aided medical procedures. IEEE Comput Gr Appl 27(5):10–14
Nolden M, Zelzer S, Seitel A, Wald D, Müller M, Franz AM, Maleike D, Fangerau M, Baumhauer M, Maier-Hein L, Maier-Hein KH, Meinzer HP, Wolf I (2013) The medical imaging interaction toolkit: challenges and advances. Int J Comput Assist Radiol Surg 8(4):607–620
Qiu D, May S, Nüchter A (2009) GPU-accelerated nearest neighbor search for 3D registration. In: Proceedings of the 7th international conference on computer vision systems. Springer, pp 194–203
Roberts IS, Benamore RE, Benbow EW, Lee SH, Harris JN, Jackson A, Mallett S, Patankar T, Peebles C, Roobottom C, Trail ZC (2012) Post-mortem imaging as an alternative to autopsy in the diagnosis of adult deaths: a validation study. Lancet 379(9811):136–142
Rusu RB, Blodow N, Beetz M (2009) Fast point feature histograms (FPFH) for 3D registration. In: IEEE international conference on robotics and automation. IEEE, pp 3212–3217
Seeling P, Fitzek FH, Ertli G, Pulipaka A, Reisslein M (2010) Video network traffic and quality comparison of VP8 and H.264 SVC. In: Proceedings of the 3rd workshop on mobile video delivery. ACM, pp 33–37
Seitel A, Engel M, Sommer CM, Radeleff BA, Essert-Villard C, Baegert C, Fangerau M, Fritzsche KH, Yung K, Meinzer HP (2011) Computer-assisted trajectory planning for percutaneous needle insertions. Med Phys 38(6):3246–3259
Seitel A, Yung K, Mersmann S, Kilgus T, Groch A, dos Santos TR, Franz AM, Nolden M, Meinzer HP, Maier-Hein L (2012) MITK-ToF: range data within MITK. Int J Comput Assist Radiol Surg 7(1):87–96
Sugimoto M, Yasuda H, Koda K, Suzuki M, Yamazaki M, Tezuka T, Kosugi C, Higuchi R, Watayo Y, Yagawa Y, Uemura S, Tsuchiya H, Azuma T (2010) Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery. J Hepato Biliary Pancreat Sci 17(5):629–636
Thali MJ, Braun M, Wirth J, Vock P, Dirnhofer R (2003) 3D surface and body documentation in forensic medicine: 3-D/CAD photogrammetry merged with 3D radiological scanning. J Forensic Sci 48(6):1356–1365
Thali MJ, Dirnhofer R, Vock P (2009) The virtopsy approach: 3D optical and radiological scanning and reconstruction in forensic medicine. CRC Press, Boca Raton
Urschler M, Höller J, Bornik A, Paul T, Giretzlehner M, Bischof H, Yen K, Scheurer E (2014) Intuitive presentation of clinical forensic data using anonymous and person-specific 3d reference manikins. Forensic Sci Int 241:155–166
Valette S, Chassery JM, Prost R (2008) Generic remeshing of 3D triangular meshes with metric-dependent discrete voronoi diagrams. IEEE Trans Vis Comput Gr 14(2):369–381
Yang J, Li H, Jia Y (2013) Go-ICP: solving 3D registration efficiently and globally optimally. In: International conference on computer vision. IEEE, pp 1457–1464
Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334
Zhu E, Masiello I, Hadadgar A, Zary N (2014) Augmented reality in healthcare education: an integrative review. PeerJ PrePrints 2:1–22
Acknowledgments
This study was performed as part of the Research Training Group 1126 Intelligente Chirurgie and project PD 15577, both funded by the German Research Foundation (DFG). We further thank all staff—the Division of Radiology of the German Cancer Research Center and the Institute for Anatomy and Cell Biology in Heidelberg—who supported the phantom and postmortem experiments. Thanks go to our intern, Simon Weiser, for assisting in postmortem experiments.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 26146 KB)
Supplementary material 2 (mp4 14791 KB)
Supplementary material 3 (mp4 66602 KB)
Rights and permissions
About this article
Cite this article
Kilgus, T., Heim, E., Haase, S. et al. Mobile markerless augmented reality and its application in forensic medicine. Int J CARS 10, 573–586 (2015). https://doi.org/10.1007/s11548-014-1106-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-014-1106-9