ABSTRACT
We present a multimodal medical 3D image system for radiologists in an virtual reality (VR) environment. Users can walk freely inside the virtual room and interact with the system using speech, going through patient records, and manipulate 3D image data with hand gestures. Medical images are retrieved from the hospital's Picture and Archiving System (PACS) and displayed as 3D objects inside VR. Our system incorporates a dialogue-based decision support system for treatments. A central supervised patient database provides input to our predictive model and allows us, first, to add new examination reports by a pen-based mobile application on-the-fly, and second, to get therapy prediction results in real-time. This demo includes a visualisation of real patient records, 3D DICOM radiology image data, and real-time therapy predictions in VR.
- Tyler Ard, David M. Krum, Thai Phan, Dominique Duncan, Ryan Essex, Mark Bolas, and Arthur Toga. 2017. NIVR: Neuro Imaging in Virtual Reality. In Proceedings of Virtual Reality (VR), 2017 IEEE. IEEE, Los Angeles, CA, 465--466.Google ScholarCross Ref
- C. Esteban, O. Staeck, S. Baier, Y. Yang, and V. Tresp. 2016. Predicting Clinical Events by Combining Static and Dynamic Information Using Recurrent Neural Networks. In 2016 IEEE International Conference on Healthcare Informatics (ICHI). 93--101.Google Scholar
- Andreas Luxenburger, Alexander Prange, Mohammad Mehdi Moniri, and Daniel Sonntag. 2016. MedicaLVR: Towards Medical Remote Collaboration Using Virtual Reality. In Proceedings of the 2016 ACM UbiComp: Adjunct (UbiComp '16). ACM, New York, NY, USA, 321--324. Google ScholarDigital Library
- Robert Nesselrath. 2015. SiAM-dp: An open development platform for massively multimodal dialogue systems in cyber-physical environments. Ph.D. Dissertation. Universität des Saarlandes, Postfach 151141, 66041 Saarbrücken.Google Scholar
- Daniel Sonntag, Volker Tresp, Sonja Zillner, Alexander Cavallaro, Matthias Hammon, André Reis, Peter A. Fasching, Martin Sedlmayr, Thomas Ganslandt, Hans-Ulrich Prokosch, Klemens Budde, Danilo Schmidt, Carl Hinrichs, Thomas Wittenberg, Philipp Daumke, and Patricia G. Oppelt. 2016. The Clinical Data Intelligence Project. Informatik-Spektrum 39, 4 (2016), 290--300.Google ScholarCross Ref
- Daniel Sonntag, Markus Weber, Alexander Cavallaro, and Matthias Hammon. 2014. Integrating Digital Pens in Breast Imaging for Instant Knowledge Acquisition. AI Magazine 35, 1 (2014), 26--37.Google ScholarCross Ref
- Daniel Sonntag, Pinar Wennerberg, Paul Buitelaar, and Sonja Zillner. 2009. Pillars of Ontology Treatment in the Medical Domain. J. Cases on Inf. Techn. 11, 4 (2009), 47--73.Google ScholarCross Ref
Index Terms
Medical 3D Images in Multimodal Virtual Reality
Recommendations
Multimodal Human Machine Interactions in Virtual and Augmented Reality
Multimodal Signals: Cognitive and Algorithmic IssuesVirtual worlds are developing rapidly over the Internet. They are visited by avatars and staffed with Embodied Conversational Agents (ECAs). An avatar is a representation of a physical person. Each person controls one or several avatars and usually ...
Multimodal augmented reality: the norm rather than the exception
MVAR '16: Proceedings of the 2016 workshop on Multimodal Virtual and Augmented RealityAugmented reality (AR) is commonly seen as a technology that overlays virtual imagery onto a participant's view of the world. In line with this, most AR research is focused on what we see. In this paper, we challenge this focus on vision and make a case ...
Multimodal prop-based interaction with virtual mock-up: CAD model integration and human performance evaluation
VRST '08: Proceedings of the 2008 ACM symposium on Virtual reality software and technologyThis paper presents a methodology for the efficient integration of CAD models in a physical-based virtual reality simulation that provides the user with multi-modal feedback. User interacts with virtual mock-up using a string-based haptic interface. ...
Comments