An investigation of a healthcare management system with the use of multimodal interaction and 3D simulation: A technical note

Current healthcare applications produce a complex and inaccessible set of data that often needs to be investigated simultaneously. As such the conflicting software applications and mental effort being demanded from the user result in time‐consuming analysis and diagnosis. The purpose of this paper is to provide a prototype, interactive system for management of multiple data sets, currently used for gait analysis capturing, reconstruction and diagnosis. In summary, this work is concerned with the development of interactive information‐visualisation software that assists medical practitioners in simplifying and enhancing the retrieval, visualisation and analysis of medical data with the intention of improving the overall system leading to an improved service for the user and patient experience.

The design of the proposed system aims to combine all the related existing software currently used for gait analysis and diagnosis under one, user‐friendly package. The latter will have the capacity to offer also real‐time, three dimensional (3D) representations of all the derived data (CT, MRI, motion capture) in an interactive virtual reality (VR) environment.

It is intended that the proposed prototype solutions will enhance interactive systems for management of multiple data sets, currently used for gait analysis capturing, reconstruction and diagnosis. The derived data encapsulate a plethora of multimedia information aiming to enhance medical visualisation.

The proposed system offers simulation capacity and a VR visualisation experience, which enhances the gait analysis diagnostic process. The 3D data can be manipulated in real‐time through a novel human‐computer interface which uses multimodal interaction through the use of graphical user interfaces and gesture recognition. The system aims towards a cost‐effective, clearly presented and timely accessible system that follows a threefold approach; It entails managing the extensive amount of the daily produced medical data, combining the scattered information related to one patient in one interface with a filtering criteria to the required information, and visualising in 3D the data from different sources, in order to improve 3D mental mapping, increase productivity and consequently ameliorate quality of service and management.