skip to main content
10.1145/3180308.3180327acmconferencesArticle/Chapter ViewAbstractPublication PagesiuiConference Proceedingsconference-collections
demonstration

Medical 3D Images in Multimodal Virtual Reality

Authors Info & Claims
Published:05 March 2018Publication History

ABSTRACT

We present a multimodal medical 3D image system for radiologists in an virtual reality (VR) environment. Users can walk freely inside the virtual room and interact with the system using speech, going through patient records, and manipulate 3D image data with hand gestures. Medical images are retrieved from the hospital's Picture and Archiving System (PACS) and displayed as 3D objects inside VR. Our system incorporates a dialogue-based decision support system for treatments. A central supervised patient database provides input to our predictive model and allows us, first, to add new examination reports by a pen-based mobile application on-the-fly, and second, to get therapy prediction results in real-time. This demo includes a visualisation of real patient records, 3D DICOM radiology image data, and real-time therapy predictions in VR.

References

  1. Tyler Ard, David M. Krum, Thai Phan, Dominique Duncan, Ryan Essex, Mark Bolas, and Arthur Toga. 2017. NIVR: Neuro Imaging in Virtual Reality. In Proceedings of Virtual Reality (VR), 2017 IEEE. IEEE, Los Angeles, CA, 465--466.Google ScholarGoogle ScholarCross RefCross Ref
  2. C. Esteban, O. Staeck, S. Baier, Y. Yang, and V. Tresp. 2016. Predicting Clinical Events by Combining Static and Dynamic Information Using Recurrent Neural Networks. In 2016 IEEE International Conference on Healthcare Informatics (ICHI). 93--101.Google ScholarGoogle Scholar
  3. Andreas Luxenburger, Alexander Prange, Mohammad Mehdi Moniri, and Daniel Sonntag. 2016. MedicaLVR: Towards Medical Remote Collaboration Using Virtual Reality. In Proceedings of the 2016 ACM UbiComp: Adjunct (UbiComp '16). ACM, New York, NY, USA, 321--324. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Robert Nesselrath. 2015. SiAM-dp: An open development platform for massively multimodal dialogue systems in cyber-physical environments. Ph.D. Dissertation. Universität des Saarlandes, Postfach 151141, 66041 Saarbrücken.Google ScholarGoogle Scholar
  5. Daniel Sonntag, Volker Tresp, Sonja Zillner, Alexander Cavallaro, Matthias Hammon, André Reis, Peter A. Fasching, Martin Sedlmayr, Thomas Ganslandt, Hans-Ulrich Prokosch, Klemens Budde, Danilo Schmidt, Carl Hinrichs, Thomas Wittenberg, Philipp Daumke, and Patricia G. Oppelt. 2016. The Clinical Data Intelligence Project. Informatik-Spektrum 39, 4 (2016), 290--300.Google ScholarGoogle ScholarCross RefCross Ref
  6. Daniel Sonntag, Markus Weber, Alexander Cavallaro, and Matthias Hammon. 2014. Integrating Digital Pens in Breast Imaging for Instant Knowledge Acquisition. AI Magazine 35, 1 (2014), 26--37.Google ScholarGoogle ScholarCross RefCross Ref
  7. Daniel Sonntag, Pinar Wennerberg, Paul Buitelaar, and Sonja Zillner. 2009. Pillars of Ontology Treatment in the Medical Domain. J. Cases on Inf. Techn. 11, 4 (2009), 47--73.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Medical 3D Images in Multimodal Virtual Reality

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in
      • Published in

        cover image ACM Conferences
        IUI '18 Companion: Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion
        March 2018
        141 pages
        ISBN:9781450355711
        DOI:10.1145/3180308

        Copyright © 2018 Owner/Author

        Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 5 March 2018

        Check for updates

        Qualifiers

        • demonstration
        • Research
        • Refereed limited

        Acceptance Rates

        Overall Acceptance Rate683of2,684submissions,25%

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader